2026-02-21T14:27:00.002 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-21T14:27:00.015 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-21T14:27:00.090 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697 branch: wip-trans-cc-opt-rbm-drop-brf description: crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/rbm$/{crimson_seastore_rbm} rados} email: xxhdx1985126@gmail.com first_in_suite: false flavor: debug job_id: '62697' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-trans-cc-opt-rbm-drop-brf ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 30 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd pool default crimson: true osd: bluestore zero block detection: true crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 25 debug seastore backref: 20 debug seastore cache: 25 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 25 debug seastore odata: 25 debug seastore omap: 25 debug seastore onode: 25 debug seastore t: 25 debug seastore tm: 25 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: seastore osd op queue: debug_random osd op queue cut off: debug_random seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 57019f91dea307b3c704ce2392a5f854512b5900 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: 57019f91dea307b3c704ce2392a5f854512b5900 workunit: branch: wip-seastore-qa-log-level-up env: CRIMSON_COMPAT: '1' sha1: c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f owner: scheduled_xuxuehan@soko04.front.sepia.ceph.com priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2477 sha1: 57019f91dea307b3c704ce2392a5f854512b5900 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-seastore-qa-log-level-up suite_path: /home/teuthworker/src/github.com_xxhdx1985126_ceph_c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f/qa suite_relpath: qa suite_repo: https://github.com/xxhdx1985126/ceph.git suite_sha1: c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f targets: trial198.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBhqKVVg8S9YFBk7J8qT75wRENCUl+fIgkPYS9SpXCxZlsytpav9rmVXW/5ICZ5bD/g3G79EDa9+3al5ZV8NWg= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: global: null osd: crimson cpu num: 2 osd min pg log entries: 5 log-ignorelist: - but it is still running - overall HEALTH_ pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: client.0: - cephtool/test_crimson_dump_metrics.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-21_14:23:14 tube: trial user: xuxuehan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-21T14:27:00.091 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_xxhdx1985126_ceph_c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f/qa; will attempt to use it 2026-02-21T14:27:00.091 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_xxhdx1985126_ceph_c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f/qa/tasks 2026-02-21T14:27:00.091 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-21T14:27:00.092 INFO:teuthology.task.internal:Checking packages... 2026-02-21T14:27:00.093 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '57019f91dea307b3c704ce2392a5f854512b5900' 2026-02-21T14:27:00.093 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-21T14:27:00.093 INFO:teuthology.packaging:ref: None 2026-02-21T14:27:00.093 INFO:teuthology.packaging:tag: None 2026-02-21T14:27:00.093 INFO:teuthology.packaging:branch: wip-trans-cc-opt-rbm-drop-brf 2026-02-21T14:27:00.093 INFO:teuthology.packaging:sha1: 57019f91dea307b3c704ce2392a5f854512b5900 2026-02-21T14:27:00.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-trans-cc-opt-rbm-drop-brf 2026-02-21T14:27:00.212 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5076.g57019f91 2026-02-21T14:27:00.213 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-21T14:27:00.226 INFO:teuthology.task.internal:no buildpackages task found 2026-02-21T14:27:00.227 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-21T14:27:00.239 INFO:teuthology.task.internal:Saving configuration 2026-02-21T14:27:00.251 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-21T14:27:00.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-21T14:27:00.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial198.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-21 14:23:25.581493', 'locked_by': 'scheduled_xuxuehan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBhqKVVg8S9YFBk7J8qT75wRENCUl+fIgkPYS9SpXCxZlsytpav9rmVXW/5ICZ5bD/g3G79EDa9+3al5ZV8NWg='} 2026-02-21T14:27:00.321 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-21T14:27:00.332 INFO:teuthology.task.internal:roles: ubuntu@trial198.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-21T14:27:00.332 INFO:teuthology.run_tasks:Running task console_log... 2026-02-21T14:27:00.407 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x77193f33e340>, signals=[15]) 2026-02-21T14:27:00.407 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-21T14:27:00.418 INFO:teuthology.task.internal:Opening connections... 2026-02-21T14:27:00.418 DEBUG:teuthology.task.internal:connecting to ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:27:00.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial198.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-21T14:27:00.481 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-21T14:27:00.492 DEBUG:teuthology.orchestra.run.trial198:> uname -m 2026-02-21T14:27:00.558 INFO:teuthology.orchestra.run.trial198.stdout:x86_64 2026-02-21T14:27:00.558 DEBUG:teuthology.orchestra.run.trial198:> cat /etc/os-release 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:NAME="CentOS Stream" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:VERSION="9" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:ID="centos" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:ID_LIKE="rhel fedora" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:VERSION_ID="9" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:PLATFORM_ID="platform:el9" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:ANSI_COLOR="0;31" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:LOGO="fedora-logo-icon" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-21T14:27:00.613 INFO:teuthology.orchestra.run.trial198.stdout:HOME_URL="https://centos.org/" 2026-02-21T14:27:00.614 INFO:teuthology.orchestra.run.trial198.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-21T14:27:00.614 INFO:teuthology.orchestra.run.trial198.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-21T14:27:00.614 INFO:teuthology.orchestra.run.trial198.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-21T14:27:00.614 INFO:teuthology.lock.ops:Updating trial198.front.sepia.ceph.com on lock server 2026-02-21T14:27:00.670 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-21T14:27:00.682 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-21T14:27:00.694 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-21T14:27:00.694 DEBUG:teuthology.orchestra.run.trial198:> test '!' -e /home/ubuntu/cephtest 2026-02-21T14:27:00.709 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-21T14:27:00.720 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-21T14:27:00.720 DEBUG:teuthology.orchestra.run.trial198:> test -z $(ls -A /var/lib/ceph) 2026-02-21T14:27:00.766 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-21T14:27:00.853 INFO:teuthology.run_tasks:Running task kernel... 2026-02-21T14:27:00.874 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-21T14:27:00.874 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-21T14:27:00.874 DEBUG:teuthology.orchestra.run.trial198:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-21T14:27:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:27:00.889 DEBUG:teuthology.orchestra.run.trial198:> uname -r 2026-02-21T14:27:00.945 INFO:teuthology.orchestra.run.trial198.stdout:5.14.0-681.el9.x86_64 2026-02-21T14:27:00.946 INFO:teuthology.task.kernel:Running kernel on trial198: 5.14.0-681.el9.x86_64 2026-02-21T14:27:00.946 DEBUG:teuthology.orchestra.run.trial198:> sudo yum install -y kernel 2026-02-21T14:27:02.026 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-21T14:27:04.624 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-21T14:27:08.384 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 7.8 MB 00:01 2026-02-21T14:27:09.559 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 20 kB 00:00 2026-02-21T14:27:10.499 INFO:teuthology.orchestra.run.trial198.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-21T14:27:13.665 INFO:teuthology.orchestra.run.trial198.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-21T14:27:14.667 INFO:teuthology.orchestra.run.trial198.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-21T14:27:14.667 INFO:teuthology.orchestra.run.trial198.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-21T14:27:14.686 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:27:14.687 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:27:14.687 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:27:14.710 DEBUG:teuthology.orchestra.run.trial198:> echo no | sudo yum reinstall kernel || true 2026-02-21T14:27:15.013 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:02 ago on Sat 21 Feb 2026 02:27:13 PM UTC. 2026-02-21T14:27:15.093 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:27:15.094 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:27:15.094 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-21T14:27:15.094 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:27:15.094 INFO:teuthology.orchestra.run.trial198.stdout:Reinstalling: 2026-02-21T14:27:15.094 INFO:teuthology.orchestra.run.trial198.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-21T14:27:15.094 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:27:15.094 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:27:15.095 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:27:15.095 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:27:15.095 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 803 k 2026-02-21T14:27:15.095 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 0 2026-02-21T14:27:15.095 INFO:teuthology.orchestra.run.trial198.stderr:Operation aborted. 2026-02-21T14:27:15.114 INFO:teuthology.orchestra.run.trial198.stdout:Is this ok [y/N]: 2026-02-21T14:27:15.115 DEBUG:teuthology.orchestra.run.trial198:> sudo yum reinstall -y kernel || true 2026-02-21T14:27:15.409 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:02 ago on Sat 21 Feb 2026 02:27:13 PM UTC. 2026-02-21T14:27:15.490 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:27:15.491 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:27:15.491 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-21T14:27:15.491 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:27:15.491 INFO:teuthology.orchestra.run.trial198.stdout:Reinstalling: 2026-02-21T14:27:15.491 INFO:teuthology.orchestra.run.trial198.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-21T14:27:15.492 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:27:15.492 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:27:15.492 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:27:15.492 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:27:15.492 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 803 k 2026-02-21T14:27:15.492 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 0 2026-02-21T14:27:15.492 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:27:16.486 INFO:teuthology.orchestra.run.trial198.stdout:kernel-5.14.0-681.el9.x86_64.rpm 865 kB/s | 803 kB 00:00 2026-02-21T14:27:16.486 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:27:16.486 INFO:teuthology.orchestra.run.trial198.stdout:Total 807 kB/s | 803 kB 00:00 2026-02-21T14:27:16.496 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:27:16.529 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:27:16.529 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:27:16.533 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:27:16.534 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:27:16.557 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:27:16.561 INFO:teuthology.orchestra.run.trial198.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-21T14:27:16.561 INFO:teuthology.orchestra.run.trial198.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-21T14:27:16.675 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-21T14:27:16.675 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-21T14:27:16.732 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-21T14:27:16.732 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:27:16.732 INFO:teuthology.orchestra.run.trial198.stdout:Reinstalled: 2026-02-21T14:27:16.732 INFO:teuthology.orchestra.run.trial198.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-21T14:27:16.732 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:27:16.732 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:27:16.759 DEBUG:teuthology.orchestra.run.trial198:> rpm -q kernel | sort -rV | head -n 1 2026-02-21T14:27:16.783 INFO:teuthology.orchestra.run.trial198.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-21T14:27:16.784 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-21T14:27:16.784 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-21T14:27:16.785 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-21T14:27:16.785 DEBUG:teuthology.orchestra.run.trial198:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-21T14:27:16.848 INFO:teuthology.orchestra.run.trial198.stdout:ttyS1 2026-02-21T14:27:16.869 DEBUG:teuthology.parallel:result is None 2026-02-21T14:27:16.869 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-21T14:27:16.881 INFO:teuthology.task.internal:Creating test directory... 2026-02-21T14:27:16.882 DEBUG:teuthology.orchestra.run.trial198:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-21T14:27:16.925 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-21T14:27:16.936 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-21T14:27:16.950 INFO:teuthology.task.internal:Creating archive directory... 2026-02-21T14:27:16.950 DEBUG:teuthology.orchestra.run.trial198:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-21T14:27:16.986 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-21T14:27:16.998 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-21T14:27:16.999 DEBUG:teuthology.orchestra.run.trial198:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-21T14:27:17.046 INFO:teuthology.orchestra.run.trial198.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T14:27:17.055 INFO:teuthology.orchestra.run.trial198.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T14:27:17.056 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-21T14:27:17.069 INFO:teuthology.task.internal:Configuring sudo... 2026-02-21T14:27:17.069 DEBUG:teuthology.orchestra.run.trial198:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-21T14:27:17.120 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-21T14:27:17.137 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-21T14:27:17.137 DEBUG:teuthology.orchestra.run.trial198:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-21T14:27:17.177 DEBUG:teuthology.orchestra.run.trial198:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T14:27:17.340 DEBUG:teuthology.orchestra.run.trial198:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T14:27:17.363 DEBUG:teuthology.orchestra.run.trial198:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T14:27:17.512 DEBUG:teuthology.orchestra.run.trial198:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T14:27:17.534 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:27:17.534 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-21T14:27:17.597 DEBUG:teuthology.orchestra.run.trial198:> sudo service rsyslog restart 2026-02-21T14:27:17.665 INFO:teuthology.orchestra.run.trial198.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-21T14:27:18.094 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-21T14:27:18.108 INFO:teuthology.task.internal:Starting timer... 2026-02-21T14:27:18.108 INFO:teuthology.run_tasks:Running task pcp... 2026-02-21T14:27:18.130 INFO:teuthology.run_tasks:Running task selinux... 2026-02-21T14:27:18.148 DEBUG:teuthology.orchestra.run.trial198:> sudo service auditd rotate 2026-02-21T14:27:18.177 INFO:teuthology.orchestra.run.trial198.stdout:Rotating logs: 2026-02-21T14:27:18.179 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-21T14:27:18.179 DEBUG:teuthology.orchestra.run.trial198:> /usr/sbin/getenforce 2026-02-21T14:27:18.236 INFO:teuthology.orchestra.run.trial198.stdout:Permissive 2026-02-21T14:27:18.236 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial198.front.sepia.ceph.com': 'permissive'} 2026-02-21T14:27:18.237 DEBUG:teuthology.orchestra.run.trial198:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-21T14:27:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:27:18.302 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-21T14:27:18.303 DEBUG:teuthology.orchestra.run.trial198:> sudo /usr/sbin/setenforce permissive 2026-02-21T14:27:18.367 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-21T14:27:18.384 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-21T14:27:18.396 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-21T14:27:18.397 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial198.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-21T14:29:50.562 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial198.front.sepia.ceph.com')] 2026-02-21T14:29:50.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial198.front.sepia.ceph.com' 2026-02-21T14:29:50.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial198.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-21T14:29:50.638 DEBUG:teuthology.orchestra.run.trial198:> true 2026-02-21T14:29:50.719 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial198.front.sepia.ceph.com' 2026-02-21T14:29:50.719 INFO:teuthology.run_tasks:Running task clock... 2026-02-21T14:29:50.734 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-21T14:29:50.734 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-21T14:29:50.734 DEBUG:teuthology.orchestra.run.trial198:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-21T14:29:50.786 INFO:teuthology.orchestra.run.trial198.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-21T14:29:50.798 INFO:teuthology.orchestra.run.trial198.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-21T14:29:50.820 INFO:teuthology.orchestra.run.trial198.stderr:sudo: ntpd: command not found 2026-02-21T14:29:50.829 INFO:teuthology.orchestra.run.trial198.stdout:506 Cannot talk to daemon 2026-02-21T14:29:50.841 INFO:teuthology.orchestra.run.trial198.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-21T14:29:50.854 INFO:teuthology.orchestra.run.trial198.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-21T14:29:50.902 INFO:teuthology.orchestra.run.trial198.stderr:bash: line 1: ntpq: command not found 2026-02-21T14:29:50.928 INFO:teuthology.orchestra.run.trial198.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-21T14:29:50.928 INFO:teuthology.orchestra.run.trial198.stdout:=============================================================================== 2026-02-21T14:29:50.928 INFO:teuthology.orchestra.run.trial198.stdout:^? us.gnoppix.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T14:29:50.928 INFO:teuthology.orchestra.run.trial198.stdout:^? arm1.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T14:29:50.928 INFO:teuthology.orchestra.run.trial198.stdout:^? 172-233-189-68.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T14:29:50.928 INFO:teuthology.orchestra.run.trial198.stdout:^? 172-233-157-223.ip.linod> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T14:29:50.928 INFO:teuthology.run_tasks:Running task install... 2026-02-21T14:29:50.940 DEBUG:teuthology.task.install:project ceph 2026-02-21T14:29:50.940 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900'}} 2026-02-21T14:29:50.940 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900'} 2026-02-21T14:29:50.940 INFO:teuthology.task.install:Using flavor: debug 2026-02-21T14:29:50.944 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-21T14:29:50.944 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-21T14:29:50.944 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900', 'tag': None, 'wait_for_package': False} 2026-02-21T14:29:50.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=57019f91dea307b3c704ce2392a5f854512b5900 2026-02-21T14:29:51.064 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/ 2026-02-21T14:29:51.064 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5076.g57019f91 2026-02-21T14:29:51.188 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-21T14:29:51.188 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:29:51.189 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-21T14:29:51.212 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-21T14:29:51.212 DEBUG:teuthology.orchestra.run.trial198:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/57019f91dea307b3c704ce2392a5f854512b5900/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-21T14:29:51.278 DEBUG:teuthology.orchestra.run.trial198:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-21T14:29:51.350 DEBUG:teuthology.orchestra.run.trial198:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-21T14:29:51.414 INFO:teuthology.orchestra.run.trial198.stdout:check_obsoletes = 1 2026-02-21T14:29:51.415 DEBUG:teuthology.orchestra.run.trial198:> sudo yum clean all 2026-02-21T14:29:51.572 INFO:teuthology.orchestra.run.trial198.stdout:41 files removed 2026-02-21T14:29:51.586 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-radosgw 2026-02-21T14:29:51.866 INFO:teuthology.orchestra.run.trial198.stdout:ceph packages for x86_64 677 kB/s | 94 kB 00:00 2026-02-21T14:29:52.014 INFO:teuthology.orchestra.run.trial198.stdout:ceph noarch packages 198 kB/s | 24 kB 00:00 2026-02-21T14:29:52.159 INFO:teuthology.orchestra.run.trial198.stdout:ceph source packages 17 kB/s | 2.1 kB 00:00 2026-02-21T14:29:52.910 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-21T14:29:56.307 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 27 MB 00:02 2026-02-21T14:29:59.591 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 7.8 MB 00:00 2026-02-21T14:30:00.447 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - Extras packages 169 kB/s | 20 kB 00:00 2026-02-21T14:30:01.160 INFO:teuthology.orchestra.run.trial198.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2026-02-21T14:30:04.338 INFO:teuthology.orchestra.run.trial198.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-21T14:30:05.534 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: ceph-radosgw x86_64 2:20.3.0-5076.g57019f91.el9 ceph 11 M 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout:Upgrading: 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: librados2 x86_64 2:20.3.0-5076.g57019f91.el9 ceph 3.3 M 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: librbd1 x86_64 2:20.3.0-5076.g57019f91.el9 ceph 2.7 M 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: ceph-base x86_64 2:20.3.0-5076.g57019f91.el9 ceph 5.1 M 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: ceph-common x86_64 2:20.3.0-5076.g57019f91.el9 ceph 21 M 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: ceph-selinux x86_64 2:20.3.0-5076.g57019f91.el9 ceph 25 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs2 x86_64 2:20.3.0-5076.g57019f91.el9 ceph 859 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: libradosstriper1 x86_64 2:20.3.0-5076.g57019f91.el9 ceph 263 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: librgw2 x86_64 2:20.3.0-5076.g57019f91.el9 ceph 5.6 M 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-argparse x86_64 2:20.3.0-5076.g57019f91.el9 ceph 45 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-common x86_64 2:20.3.0-5076.g57019f91.el9 ceph 184 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: python3-cephfs x86_64 2:20.3.0-5076.g57019f91.el9 ceph 204 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: python3-rados x86_64 2:20.3.0-5076.g57019f91.el9 ceph 324 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: python3-rbd x86_64 2:20.3.0-5076.g57019f91.el9 ceph 305 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: python3-rgw x86_64 2:20.3.0-5076.g57019f91.el9 ceph 100 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-21T14:30:05.536 INFO:teuthology.orchestra.run.trial198.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout:Installing weak dependencies: 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-daemon x86_64 2:20.3.0-5076.g57019f91.el9 ceph 35 k 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5076.g57019f91.el9 ceph 24 k 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout:Install 35 Packages 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout:Upgrade 2 Packages 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 61 M 2026-02-21T14:30:05.537 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:30:06.618 INFO:teuthology.orchestra.run.trial198.stdout:(1/37): ceph-base-20.3.0-5076.g57019f91.el9.x86 14 MB/s | 5.1 MB 00:00 2026-02-21T14:30:06.666 INFO:teuthology.orchestra.run.trial198.stdout:(2/37): ceph-selinux-20.3.0-5076.g57019f91.el9. 527 kB/s | 25 kB 00:00 2026-02-21T14:30:06.715 INFO:teuthology.orchestra.run.trial198.stdout:(3/37): libcephfs-daemon-20.3.0-5076.g57019f91. 712 kB/s | 35 kB 00:00 2026-02-21T14:30:06.768 INFO:teuthology.orchestra.run.trial198.stdout:(4/37): libcephfs-proxy2-20.3.0-5076.g57019f91. 459 kB/s | 24 kB 00:00 2026-02-21T14:30:06.841 INFO:teuthology.orchestra.run.trial198.stdout:(5/37): libcephfs2-20.3.0-5076.g57019f91.el9.x8 12 MB/s | 859 kB 00:00 2026-02-21T14:30:06.898 INFO:teuthology.orchestra.run.trial198.stdout:(6/37): libradosstriper1-20.3.0-5076.g57019f91. 4.5 MB/s | 263 kB 00:00 2026-02-21T14:30:06.945 INFO:teuthology.orchestra.run.trial198.stdout:(7/37): ceph-common-20.3.0-5076.g57019f91.el9.x 30 MB/s | 21 MB 00:00 2026-02-21T14:30:06.977 INFO:teuthology.orchestra.run.trial198.stdout:(8/37): python3-ceph-argparse-20.3.0-5076.g5701 1.4 MB/s | 45 kB 00:00 2026-02-21T14:30:07.023 INFO:teuthology.orchestra.run.trial198.stdout:(9/37): python3-ceph-common-20.3.0-5076.g57019f 3.9 MB/s | 184 kB 00:00 2026-02-21T14:30:07.078 INFO:teuthology.orchestra.run.trial198.stdout:(10/37): python3-cephfs-20.3.0-5076.g57019f91.e 3.6 MB/s | 204 kB 00:00 2026-02-21T14:30:07.127 INFO:teuthology.orchestra.run.trial198.stdout:(11/37): librgw2-20.3.0-5076.g57019f91.el9.x86_ 24 MB/s | 5.6 MB 00:00 2026-02-21T14:30:07.131 INFO:teuthology.orchestra.run.trial198.stdout:(12/37): python3-rados-20.3.0-5076.g57019f91.el 6.0 MB/s | 324 kB 00:00 2026-02-21T14:30:07.173 INFO:teuthology.orchestra.run.trial198.stdout:(13/37): python3-rbd-20.3.0-5076.g57019f91.el9. 6.6 MB/s | 305 kB 00:00 2026-02-21T14:30:07.175 INFO:teuthology.orchestra.run.trial198.stdout:(14/37): python3-rgw-20.3.0-5076.g57019f91.el9. 2.2 MB/s | 100 kB 00:00 2026-02-21T14:30:07.210 INFO:teuthology.orchestra.run.trial198.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 906 kB/s | 33 kB 00:00 2026-02-21T14:30:07.241 INFO:teuthology.orchestra.run.trial198.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 3.0 MB/s | 93 kB 00:00 2026-02-21T14:30:07.281 INFO:teuthology.orchestra.run.trial198.stdout:(17/37): ceph-radosgw-20.3.0-5076.g57019f91.el9 11 MB/s | 11 MB 00:01 2026-02-21T14:30:07.287 INFO:teuthology.orchestra.run.trial198.stdout:(18/37): nvme-cli-2.16-1.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2026-02-21T14:30:07.860 INFO:teuthology.orchestra.run.trial198.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 168 kB/s | 104 kB 00:00 2026-02-21T14:30:07.934 INFO:teuthology.orchestra.run.trial198.stdout:(20/37): libnbd-1.20.3-4.el9.x86_64.rpm 251 kB/s | 164 kB 00:00 2026-02-21T14:30:07.962 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-21T14:30:07.962 INFO:teuthology.orchestra.run.trial198.stdout:(21/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 237 kB/s | 160 kB 00:00 2026-02-21T14:30:08.033 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-21T14:30:08.066 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-21T14:30:08.367 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-21T14:30:08.402 INFO:teuthology.orchestra.run.trial198.stdout:(22/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 83 kB/s | 45 kB 00:00 2026-02-21T14:30:08.410 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-21T14:30:08.440 INFO:teuthology.orchestra.run.trial198.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 612 kB/s | 292 kB 00:00 2026-02-21T14:30:08.446 INFO:teuthology.orchestra.run.trial198.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-21T14:30:08.468 INFO:teuthology.orchestra.run.trial198.stdout:(24/37): librdkafka-1.6.1-102.el9.x86_64.rpm 1.2 MB/s | 662 kB 00:00 2026-02-21T14:30:08.479 INFO:teuthology.orchestra.run.trial198.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 5.6 MB/s | 66 kB 00:00 2026-02-21T14:30:08.553 INFO:teuthology.orchestra.run.trial198.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.1 MB/s | 308 kB 00:00 2026-02-21T14:30:08.608 INFO:teuthology.orchestra.run.trial198.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 81 MB/s | 4.4 MB 00:00 2026-02-21T14:30:08.618 INFO:teuthology.orchestra.run.trial198.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 2.4 MB/s | 25 kB 00:00 2026-02-21T14:30:08.628 INFO:teuthology.orchestra.run.trial198.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 4.9 MB/s | 49 kB 00:00 2026-02-21T14:30:08.638 INFO:teuthology.orchestra.run.trial198.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 6.9 MB/s | 67 kB 00:00 2026-02-21T14:30:08.651 INFO:teuthology.orchestra.run.trial198.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 63 MB/s | 838 kB 00:00 2026-02-21T14:30:08.661 INFO:teuthology.orchestra.run.trial198.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 19 MB/s | 191 kB 00:00 2026-02-21T14:30:08.677 INFO:teuthology.orchestra.run.trial198.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 102 MB/s | 1.6 MB 00:00 2026-02-21T14:30:08.865 INFO:teuthology.orchestra.run.trial198.stdout:(34/37): qatlib-service-25.08.0-2.el9.x86_64.rp 87 kB/s | 37 kB 00:00 2026-02-21T14:30:09.136 INFO:teuthology.orchestra.run.trial198.stdout:(35/37): qatlib-25.08.0-2.el9.x86_64.rpm 326 kB/s | 240 kB 00:00 2026-02-21T14:30:09.211 INFO:teuthology.orchestra.run.trial198.stdout:(36/37): librados2-20.3.0-5076.g57019f91.el9.x8 6.2 MB/s | 3.3 MB 00:00 2026-02-21T14:30:09.329 INFO:teuthology.orchestra.run.trial198.stdout:(37/37): librbd1-20.3.0-5076.g57019f91.el9.x86_ 5.8 MB/s | 2.7 MB 00:00 2026-02-21T14:30:09.333 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:30:09.333 INFO:teuthology.orchestra.run.trial198.stdout:Total 16 MB/s | 61 MB 00:03 2026-02-21T14:30:09.433 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:30:09.453 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:30:09.453 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:30:09.704 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:30:09.704 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:30:10.004 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:30:10.142 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-21T14:30:10.147 INFO:teuthology.orchestra.run.trial198.stdout: Upgrading : librados2-2:20.3.0-5076.g57019f91.el9.x86_64 2/39 2026-02-21T14:30:10.199 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librados2-2:20.3.0-5076.g57019f91.el9.x86_64 2/39 2026-02-21T14:30:10.200 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephfs2-2:20.3.0-5076.g57019f91.el9.x86_64 3/39 2026-02-21T14:30:10.213 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephfs2-2:20.3.0-5076.g57019f91.el9.x86_64 3/39 2026-02-21T14:30:10.215 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephfs-daemon-2:20.3.0-5076.g57019f91.el9.x86_6 4/39 2026-02-21T14:30:10.216 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephfs-proxy2-2:20.3.0-5076.g57019f91.el9.x86_6 5/39 2026-02-21T14:30:10.243 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5076.g57019f91.el9.x86_6 5/39 2026-02-21T14:30:10.245 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rados-2:20.3.0-5076.g57019f91.el9.x86_64 6/39 2026-02-21T14:30:10.255 INFO:teuthology.orchestra.run.trial198.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-21T14:30:10.257 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-21T14:30:10.260 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-21T14:30:10.264 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-21T14:30:10.275 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-ceph-argparse-2:20.3.0-5076.g57019f91.el9. 11/39 2026-02-21T14:30:10.288 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cephfs-2:20.3.0-5076.g57019f91.el9.x86_64 12/39 2026-02-21T14:30:10.289 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libradosstriper1-2:20.3.0-5076.g57019f91.el9.x86_6 13/39 2026-02-21T14:30:10.311 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5076.g57019f91.el9.x86_6 13/39 2026-02-21T14:30:10.315 INFO:teuthology.orchestra.run.trial198.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-21T14:30:10.345 INFO:teuthology.orchestra.run.trial198.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-21T14:30:10.348 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-21T14:30:10.354 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-21T14:30:10.357 INFO:teuthology.orchestra.run.trial198.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-21T14:30:10.362 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-21T14:30:10.475 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-21T14:30:10.476 INFO:teuthology.orchestra.run.trial198.stdout: Upgrading : librbd1-2:20.3.0-5076.g57019f91.el9.x86_64 21/39 2026-02-21T14:30:10.502 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librbd1-2:20.3.0-5076.g57019f91.el9.x86_64 21/39 2026-02-21T14:30:10.504 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rbd-2:20.3.0-5076.g57019f91.el9.x86_64 22/39 2026-02-21T14:30:10.511 INFO:teuthology.orchestra.run.trial198.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-21T14:30:10.757 INFO:teuthology.orchestra.run.trial198.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-21T14:30:10.759 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librgw2-2:20.3.0-5076.g57019f91.el9.x86_64 25/39 2026-02-21T14:30:10.775 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librgw2-2:20.3.0-5076.g57019f91.el9.x86_64 25/39 2026-02-21T14:30:10.779 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rgw-2:20.3.0-5076.g57019f91.el9.x86_64 26/39 2026-02-21T14:30:10.784 INFO:teuthology.orchestra.run.trial198.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-21T14:30:10.786 INFO:teuthology.orchestra.run.trial198.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-21T14:30:10.787 INFO:teuthology.orchestra.run.trial198.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-21T14:30:10.820 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-21T14:30:10.935 INFO:teuthology.orchestra.run.trial198.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-21T14:30:10.938 INFO:teuthology.orchestra.run.trial198.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-21T14:30:11.214 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-21T14:30:11.214 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-21T14:30:11.214 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:11.259 INFO:teuthology.orchestra.run.trial198.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-21T14:30:11.261 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-ceph-common-2:20.3.0-5076.g57019f91.el9.x8 33/39 2026-02-21T14:30:12.201 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-common-2:20.3.0-5076.g57019f91.el9.x86_64 34/39 2026-02-21T14:30:12.204 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-common-2:20.3.0-5076.g57019f91.el9.x86_64 34/39 2026-02-21T14:30:12.464 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-common-2:20.3.0-5076.g57019f91.el9.x86_64 34/39 2026-02-21T14:30:12.467 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-base-2:20.3.0-5076.g57019f91.el9.x86_64 35/39 2026-02-21T14:30:12.508 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-base-2:20.3.0-5076.g57019f91.el9.x86_64 35/39 2026-02-21T14:30:12.508 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-21T14:30:12.508 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-21T14:30:12.508 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:12.512 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-selinux-2:20.3.0-5076.g57019f91.el9.x86_64 36/39 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5076.g57019f91.el9.x86_64 36/39 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /sys 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /proc 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /mnt 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /var/tmp 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /home 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /root 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout:skipping the directory /tmp 2026-02-21T14:30:16.976 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:17.467 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-radosgw-2:20.3.0-5076.g57019f91.el9.x86_64 37/39 2026-02-21T14:30:17.491 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5076.g57019f91.el9.x86_64 37/39 2026-02-21T14:30:17.491 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:30:17.491 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-21T14:30:17.491 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-21T14:30:17.491 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-21T14:30:17.491 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:17.492 INFO:teuthology.orchestra.run.trial198.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-21T14:30:17.504 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-21T14:30:17.504 INFO:teuthology.orchestra.run.trial198.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-21T14:30:17.560 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-21T14:30:17.561 INFO:teuthology.orchestra.run.trial198.stdout:Creating group 'qat' with GID 992. 2026-02-21T14:30:17.561 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:17.868 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-base-2:20.3.0-5076.g57019f91.el9.x86_64 1/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-common-2:20.3.0-5076.g57019f91.el9.x86_64 2/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-radosgw-2:20.3.0-5076.g57019f91.el9.x86_64 3/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-selinux-2:20.3.0-5076.g57019f91.el9.x86_64 4/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephfs-daemon-2:20.3.0-5076.g57019f91.el9.x86_6 5/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5076.g57019f91.el9.x86_6 6/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephfs2-2:20.3.0-5076.g57019f91.el9.x86_64 7/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libradosstriper1-2:20.3.0-5076.g57019f91.el9.x86_6 8/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librgw2-2:20.3.0-5076.g57019f91.el9.x86_64 9/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5076.g57019f91.el9. 10/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-ceph-common-2:20.3.0-5076.g57019f91.el9.x8 11/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cephfs-2:20.3.0-5076.g57019f91.el9.x86_64 12/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rados-2:20.3.0-5076.g57019f91.el9.x86_64 13/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rbd-2:20.3.0-5076.g57019f91.el9.x86_64 14/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rgw-2:20.3.0-5076.g57019f91.el9.x86_64 15/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-21T14:30:17.869 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librados2-2:20.3.0-5076.g57019f91.el9.x86_64 36/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-21T14:30:17.870 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librbd1-2:20.3.0-5076.g57019f91.el9.x86_64 38/39 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout:Upgraded: 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: librados2-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: librbd1-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: ceph-base-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: ceph-common-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: ceph-radosgw-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.934 INFO:teuthology.orchestra.run.trial198.stdout: ceph-selinux-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-daemon-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-proxy2-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs2-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libradosstriper1-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: librgw2-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-argparse-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: python3-ceph-common-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: python3-cephfs-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: python3-rados-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: python3-rbd-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: python3-rgw-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: re2-1:20211101-20.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:17.935 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:30:17.977 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-test 2026-02-21T14:30:18.288 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:14 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: ceph-test x86_64 2:20.3.0-5076.g57019f91.el9 ceph 126 M 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: libcephsqlite x86_64 2:20.3.0-5076.g57019f91.el9 ceph 153 k 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-21T14:30:18.371 INFO:teuthology.orchestra.run.trial198.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout:Install 12 Packages 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 131 M 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 977 M 2026-02-21T14:30:18.372 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:30:19.303 INFO:teuthology.orchestra.run.trial198.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 7.4 MB/s | 2.1 MB 00:00 2026-02-21T14:30:19.326 INFO:teuthology.orchestra.run.trial198.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 985 kB/s | 23 kB 00:00 2026-02-21T14:30:19.418 INFO:teuthology.orchestra.run.trial198.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.0 MB/s | 94 kB 00:00 2026-02-21T14:30:19.476 INFO:teuthology.orchestra.run.trial198.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 8.8 MB/s | 525 kB 00:00 2026-02-21T14:30:19.487 INFO:teuthology.orchestra.run.trial198.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 16 MB/s | 176 kB 00:00 2026-02-21T14:30:19.498 INFO:teuthology.orchestra.run.trial198.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 21 MB/s | 233 kB 00:00 2026-02-21T14:30:19.530 INFO:teuthology.orchestra.run.trial198.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 31 MB/s | 1.0 MB 00:00 2026-02-21T14:30:19.543 INFO:teuthology.orchestra.run.trial198.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 23 MB/s | 303 kB 00:00 2026-02-21T14:30:19.553 INFO:teuthology.orchestra.run.trial198.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 5.8 MB/s | 64 kB 00:00 2026-02-21T14:30:19.740 INFO:teuthology.orchestra.run.trial198.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 653 kB/s | 122 kB 00:00 2026-02-21T14:30:19.745 INFO:teuthology.orchestra.run.trial198.stdout:(11/12): libcephsqlite-20.3.0-5076.g57019f91.el 211 kB/s | 153 kB 00:00 2026-02-21T14:34:11.822 INFO:teuthology.orchestra.run.trial198.stdout:(12/12): ceph-test-20.3.0-5076.g57019f91.el9.x8 555 kB/s | 126 MB 03:52 2026-02-21T14:34:11.823 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:34:11.823 INFO:teuthology.orchestra.run.trial198.stdout:Total 574 kB/s | 131 MB 03:53 2026-02-21T14:34:11.889 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:34:11.900 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:34:11.900 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:34:12.332 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:34:12.333 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:34:12.795 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:34:12.804 INFO:teuthology.orchestra.run.trial198.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-21T14:34:12.812 INFO:teuthology.orchestra.run.trial198.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-21T14:34:12.817 INFO:teuthology.orchestra.run.trial198.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-21T14:34:12.821 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-21T14:34:12.824 INFO:teuthology.orchestra.run.trial198.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-21T14:34:12.828 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-21T14:34:12.829 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-21T14:34:12.844 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-21T14:34:12.846 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-21T14:34:12.856 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-21T14:34:12.866 INFO:teuthology.orchestra.run.trial198.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-21T14:34:12.867 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephsqlite-2:20.3.0-5076.g57019f91.el9.x86_64 11/12 2026-02-21T14:34:19.754 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5076.g57019f91.el9.x86_64 11/12 2026-02-21T14:34:19.772 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-test-2:20.3.0-5076.g57019f91.el9.x86_64 12/12 2026-02-21T14:34:20.592 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-test-2:20.3.0-5076.g57019f91.el9.x86_64 12/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-test-2:20.3.0-5076.g57019f91.el9.x86_64 1/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephsqlite-2:20.3.0-5076.g57019f91.el9.x86_64 2/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-21T14:34:20.593 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-21T14:34:20.594 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-21T14:34:20.647 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-21T14:34:20.647 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:34:20.647 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:34:20.647 INFO:teuthology.orchestra.run.trial198.stdout: ceph-test-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:34:20.647 INFO:teuthology.orchestra.run.trial198.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-21T14:34:20.647 INFO:teuthology.orchestra.run.trial198.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-21T14:34:20.647 INFO:teuthology.orchestra.run.trial198.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: libcephsqlite-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:34:20.648 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:34:20.688 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph 2026-02-21T14:34:20.993 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:04:17 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:34:21.078 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:34:21.081 INFO:teuthology.orchestra.run.trial198.stdout:======================================================================================= 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repository Size 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout:======================================================================================= 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout: ceph x86_64 2:20.3.0-5076.g57019f91.el9 ceph 6.5 k 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout:Installing dependencies: 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 57 k 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mds x86_64 2:20.3.0-5076.g57019f91.el9 ceph 2.0 M 2026-02-21T14:34:21.082 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr x86_64 2:20.3.0-5076.g57019f91.el9 ceph 937 k 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 293 k 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mon x86_64 2:20.3.0-5076.g57019f91.el9 ceph 4.8 M 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd x86_64 2:20.3.0-5076.g57019f91.el9 ceph 163 k 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-classic x86_64 2:20.3.0-5076.g57019f91.el9 ceph 15 M 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson x86_64 2:20.3.0-5076.g57019f91.el9 ceph 87 M 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 17 k 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: cephadm noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 1.0 M 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-21T14:34:21.083 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-21T14:34:21.084 INFO:teuthology.orchestra.run.trial198.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-21T14:34:21.085 INFO:teuthology.orchestra.run.trial198.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-21T14:34:21.086 INFO:teuthology.orchestra.run.trial198.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-21T14:34:21.087 INFO:teuthology.orchestra.run.trial198.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-21T14:34:21.088 INFO:teuthology.orchestra.run.trial198.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-21T14:34:21.089 INFO:teuthology.orchestra.run.trial198.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout:Installing weak dependencies: 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 183 k 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 15 M 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 7.4 M 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 22 k 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-rook noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 50 k 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: ceph-volume noarch 2:20.3.0-5076.g57019f91.el9 ceph-noarch 299 k 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:34:21.090 INFO:teuthology.orchestra.run.trial198.stdout:======================================================================================= 2026-02-21T14:34:21.091 INFO:teuthology.orchestra.run.trial198.stdout:Install 96 Packages 2026-02-21T14:34:21.091 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:34:21.091 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 189 M 2026-02-21T14:34:21.092 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 1.4 G 2026-02-21T14:34:21.092 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:34:21.883 INFO:teuthology.orchestra.run.trial198.stdout:(1/96): ceph-20.3.0-5076.g57019f91.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-21T14:34:23.898 INFO:teuthology.orchestra.run.trial198.stdout:(2/96): ceph-mon-20.3.0-5076.g57019f91.el9.x86_ 2.4 MB/s | 4.8 MB 00:02 2026-02-21T14:34:24.038 INFO:teuthology.orchestra.run.trial198.stdout:(3/96): ceph-osd-20.3.0-5076.g57019f91.el9.x86_ 1.1 MB/s | 163 kB 00:00 2026-02-21T14:34:25.355 INFO:teuthology.orchestra.run.trial198.stdout:(4/96): ceph-mgr-20.3.0-5076.g57019f91.el9.x86_ 249 kB/s | 937 kB 00:03 2026-02-21T14:34:28.648 INFO:teuthology.orchestra.run.trial198.stdout:(5/96): ceph-mds-20.3.0-5076.g57019f91.el9.x86_ 294 kB/s | 2.0 MB 00:07 2026-02-21T14:34:28.858 INFO:teuthology.orchestra.run.trial198.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5076.g57 272 kB/s | 57 kB 00:00 2026-02-21T14:34:29.341 INFO:teuthology.orchestra.run.trial198.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5076.g57019f91. 379 kB/s | 183 kB 00:00 2026-02-21T14:34:57.228 INFO:teuthology.orchestra.run.trial198.stdout:(8/96): ceph-osd-classic-20.3.0-5076.g57019f91. 455 kB/s | 15 MB 00:33 2026-02-21T14:35:06.729 INFO:teuthology.orchestra.run.trial198.stdout:(9/96): ceph-mgr-dashboard-20.3.0-5076.g57019f9 409 kB/s | 15 MB 00:37 2026-02-21T14:35:06.868 INFO:teuthology.orchestra.run.trial198.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5076.g57019f 159 kB/s | 22 kB 00:00 2026-02-21T14:35:08.393 INFO:teuthology.orchestra.run.trial198.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5076.g570 192 kB/s | 293 kB 00:01 2026-02-21T14:35:08.600 INFO:teuthology.orchestra.run.trial198.stdout:(12/96): ceph-mgr-rook-20.3.0-5076.g57019f91.el 242 kB/s | 50 kB 00:00 2026-02-21T14:35:08.671 INFO:teuthology.orchestra.run.trial198.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5076.g57 243 kB/s | 17 kB 00:00 2026-02-21T14:35:09.869 INFO:teuthology.orchestra.run.trial198.stdout:(14/96): ceph-volume-20.3.0-5076.g57019f91.el9. 249 kB/s | 299 kB 00:01 2026-02-21T14:35:12.924 INFO:teuthology.orchestra.run.trial198.stdout:(15/96): cephadm-20.3.0-5076.g57019f91.el9.noar 340 kB/s | 1.0 MB 00:03 2026-02-21T14:35:13.058 INFO:teuthology.orchestra.run.trial198.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.6 MB/s | 351 kB 00:00 2026-02-21T14:35:13.098 INFO:teuthology.orchestra.run.trial198.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.0 MB/s | 40 kB 00:00 2026-02-21T14:35:13.144 INFO:teuthology.orchestra.run.trial198.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 1.5 MB/s | 72 kB 00:00 2026-02-21T14:35:13.220 INFO:teuthology.orchestra.run.trial198.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 10 MB/s | 794 kB 00:00 2026-02-21T14:35:13.262 INFO:teuthology.orchestra.run.trial198.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 4.4 MB/s | 184 kB 00:00 2026-02-21T14:35:13.312 INFO:teuthology.orchestra.run.trial198.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2026-02-21T14:35:13.373 INFO:teuthology.orchestra.run.trial198.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 20 MB/s | 1.2 MB 00:00 2026-02-21T14:35:13.413 INFO:teuthology.orchestra.run.trial198.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 2.6 MB/s | 106 kB 00:00 2026-02-21T14:35:13.451 INFO:teuthology.orchestra.run.trial198.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 3.4 MB/s | 135 kB 00:00 2026-02-21T14:35:13.491 INFO:teuthology.orchestra.run.trial198.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 3.7 MB/s | 150 kB 00:00 2026-02-21T14:35:13.531 INFO:teuthology.orchestra.run.trial198.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 3.0 MB/s | 126 kB 00:00 2026-02-21T14:35:13.572 INFO:teuthology.orchestra.run.trial198.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 5.3 MB/s | 218 kB 00:00 2026-02-21T14:35:13.613 INFO:teuthology.orchestra.run.trial198.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 4.3 MB/s | 182 kB 00:00 2026-02-21T14:35:13.654 INFO:teuthology.orchestra.run.trial198.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 6.4 MB/s | 266 kB 00:00 2026-02-21T14:35:13.710 INFO:teuthology.orchestra.run.trial198.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 524 kB/s | 30 kB 00:00 2026-02-21T14:35:13.844 INFO:teuthology.orchestra.run.trial198.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 22 MB/s | 3.0 MB 00:00 2026-02-21T14:35:13.863 INFO:teuthology.orchestra.run.trial198.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 761 kB/s | 15 kB 00:00 2026-02-21T14:35:13.884 INFO:teuthology.orchestra.run.trial198.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 11 MB/s | 246 kB 00:00 2026-02-21T14:35:13.906 INFO:teuthology.orchestra.run.trial198.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 8.7 MB/s | 188 kB 00:00 2026-02-21T14:35:13.925 INFO:teuthology.orchestra.run.trial198.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 2.1 MB/s | 42 kB 00:00 2026-02-21T14:35:13.975 INFO:teuthology.orchestra.run.trial198.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 107 MB/s | 5.3 MB 00:00 2026-02-21T14:35:14.028 INFO:teuthology.orchestra.run.trial198.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 111 MB/s | 6.0 MB 00:00 2026-02-21T14:35:14.051 INFO:teuthology.orchestra.run.trial198.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 11 MB/s | 244 kB 00:00 2026-02-21T14:35:14.073 INFO:teuthology.orchestra.run.trial198.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 11 MB/s | 249 kB 00:00 2026-02-21T14:35:14.094 INFO:teuthology.orchestra.run.trial198.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 2.3 MB/s | 48 kB 00:00 2026-02-21T14:35:14.115 INFO:teuthology.orchestra.run.trial198.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 8.2 MB/s | 177 kB 00:00 2026-02-21T14:35:14.140 INFO:teuthology.orchestra.run.trial198.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 49 MB/s | 1.2 MB 00:00 2026-02-21T14:35:14.160 INFO:teuthology.orchestra.run.trial198.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.7 MB/s | 35 kB 00:00 2026-02-21T14:35:14.213 INFO:teuthology.orchestra.run.trial198.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 116 MB/s | 6.1 MB 00:00 2026-02-21T14:35:14.237 INFO:teuthology.orchestra.run.trial198.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 18 MB/s | 442 kB 00:00 2026-02-21T14:35:14.258 INFO:teuthology.orchestra.run.trial198.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 3.6 MB/s | 77 kB 00:00 2026-02-21T14:35:14.279 INFO:teuthology.orchestra.run.trial198.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 12 MB/s | 267 kB 00:00 2026-02-21T14:35:14.301 INFO:teuthology.orchestra.run.trial198.stdout:(48/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 7.1 MB/s | 159 kB 00:00 2026-02-21T14:35:14.324 INFO:teuthology.orchestra.run.trial198.stdout:(49/96): python3-pyasn1-modules-0.4.8-6.el9.noa 12 MB/s | 279 kB 00:00 2026-02-21T14:35:14.344 INFO:teuthology.orchestra.run.trial198.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.6 MB/s | 54 kB 00:00 2026-02-21T14:35:14.509 INFO:teuthology.orchestra.run.trial198.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 117 MB/s | 19 MB 00:00 2026-02-21T14:35:14.531 INFO:teuthology.orchestra.run.trial198.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.9 MB/s | 42 kB 00:00 2026-02-21T14:35:14.552 INFO:teuthology.orchestra.run.trial198.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 8.9 MB/s | 189 kB 00:00 2026-02-21T14:35:14.572 INFO:teuthology.orchestra.run.trial198.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 4.3 MB/s | 90 kB 00:00 2026-02-21T14:35:14.803 INFO:teuthology.orchestra.run.trial198.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 97 kB/s | 22 kB 00:00 2026-02-21T14:35:14.949 INFO:teuthology.orchestra.run.trial198.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 5.8 MB/s | 862 kB 00:00 2026-02-21T14:35:15.036 INFO:teuthology.orchestra.run.trial198.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 6.2 MB/s | 551 kB 00:00 2026-02-21T14:35:15.046 INFO:teuthology.orchestra.run.trial198.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 1.9 MB/s | 19 kB 00:00 2026-02-21T14:35:15.075 INFO:teuthology.orchestra.run.trial198.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 5.2 MB/s | 151 kB 00:00 2026-02-21T14:35:15.120 INFO:teuthology.orchestra.run.trial198.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 12 MB/s | 548 kB 00:00 2026-02-21T14:35:15.132 INFO:teuthology.orchestra.run.trial198.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 2.5 MB/s | 29 kB 00:00 2026-02-21T14:35:15.148 INFO:teuthology.orchestra.run.trial198.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 3.6 MB/s | 60 kB 00:00 2026-02-21T14:35:15.167 INFO:teuthology.orchestra.run.trial198.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.2 MB/s | 43 kB 00:00 2026-02-21T14:35:15.172 INFO:teuthology.orchestra.run.trial198.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 7.0 MB/s | 32 kB 00:00 2026-02-21T14:35:15.176 INFO:teuthology.orchestra.run.trial198.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 3.5 MB/s | 14 kB 00:00 2026-02-21T14:35:15.190 INFO:teuthology.orchestra.run.trial198.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 12 MB/s | 173 kB 00:00 2026-02-21T14:35:15.214 INFO:teuthology.orchestra.run.trial198.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 15 MB/s | 358 kB 00:00 2026-02-21T14:35:15.239 INFO:teuthology.orchestra.run.trial198.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 9.9 MB/s | 254 kB 00:00 2026-02-21T14:35:15.328 INFO:teuthology.orchestra.run.trial198.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 23 MB/s | 2.0 MB 00:00 2026-02-21T14:35:15.338 INFO:teuthology.orchestra.run.trial198.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 14 MB/s | 144 kB 00:00 2026-02-21T14:35:15.347 INFO:teuthology.orchestra.run.trial198.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 16 MB/s | 139 kB 00:00 2026-02-21T14:35:15.352 INFO:teuthology.orchestra.run.trial198.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 12 MB/s | 56 kB 00:00 2026-02-21T14:35:15.356 INFO:teuthology.orchestra.run.trial198.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 2.7 MB/s | 11 kB 00:00 2026-02-21T14:35:15.360 INFO:teuthology.orchestra.run.trial198.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 4.4 MB/s | 18 kB 00:00 2026-02-21T14:35:15.364 INFO:teuthology.orchestra.run.trial198.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 6.1 MB/s | 23 kB 00:00 2026-02-21T14:35:15.377 INFO:teuthology.orchestra.run.trial198.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 1.4 MB/s | 20 kB 00:00 2026-02-21T14:35:15.381 INFO:teuthology.orchestra.run.trial198.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 5.1 MB/s | 19 kB 00:00 2026-02-21T14:35:15.391 INFO:teuthology.orchestra.run.trial198.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 2.6 MB/s | 26 kB 00:00 2026-02-21T14:35:15.428 INFO:teuthology.orchestra.run.trial198.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 28 MB/s | 1.0 MB 00:00 2026-02-21T14:35:15.439 INFO:teuthology.orchestra.run.trial198.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 7.2 MB/s | 79 kB 00:00 2026-02-21T14:35:15.447 INFO:teuthology.orchestra.run.trial198.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 10 MB/s | 86 kB 00:00 2026-02-21T14:35:15.452 INFO:teuthology.orchestra.run.trial198.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 12 MB/s | 58 kB 00:00 2026-02-21T14:35:15.462 INFO:teuthology.orchestra.run.trial198.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 1.6 MB/s | 16 kB 00:00 2026-02-21T14:35:15.474 INFO:teuthology.orchestra.run.trial198.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 7.4 MB/s | 90 kB 00:00 2026-02-21T14:35:15.479 INFO:teuthology.orchestra.run.trial198.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 6.6 MB/s | 31 kB 00:00 2026-02-21T14:35:15.498 INFO:teuthology.orchestra.run.trial198.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 9.5 MB/s | 188 kB 00:00 2026-02-21T14:35:15.515 INFO:teuthology.orchestra.run.trial198.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2026-02-21T14:35:15.523 INFO:teuthology.orchestra.run.trial198.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 15 MB/s | 125 kB 00:00 2026-02-21T14:35:15.528 INFO:teuthology.orchestra.run.trial198.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 7.3 MB/s | 36 kB 00:00 2026-02-21T14:35:15.541 INFO:teuthology.orchestra.run.trial198.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 6.4 MB/s | 86 kB 00:00 2026-02-21T14:35:15.551 INFO:teuthology.orchestra.run.trial198.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 9.1 MB/s | 90 kB 00:00 2026-02-21T14:35:15.566 INFO:teuthology.orchestra.run.trial198.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 3.1 MB/s | 48 kB 00:00 2026-02-21T14:35:15.572 INFO:teuthology.orchestra.run.trial198.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 3.7 MB/s | 22 kB 00:00 2026-02-21T14:35:15.581 INFO:teuthology.orchestra.run.trial198.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 2.3 MB/s | 20 kB 00:00 2026-02-21T14:35:18.243 INFO:teuthology.orchestra.run.trial198.stdout:(95/96): ceph-mgr-diskprediction-local-20.3.0-5 361 kB/s | 7.4 MB 00:21 2026-02-21T14:39:24.808 INFO:teuthology.orchestra.run.trial198.stdout:(96/96): ceph-osd-crimson-20.3.0-5076.g57019f91 296 kB/s | 87 MB 04:59 2026-02-21T14:39:24.810 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:39:24.811 INFO:teuthology.orchestra.run.trial198.stdout:Total 637 kB/s | 189 MB 05:03 2026-02-21T14:39:25.176 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:39:25.215 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:39:25.215 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:39:25.908 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:39:25.909 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:39:26.644 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:39:26.646 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-21T14:39:26.650 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-21T14:39:26.671 INFO:teuthology.orchestra.run.trial198.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-21T14:39:26.704 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-21T14:39:26.710 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-21T14:39:26.714 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-21T14:39:26.718 INFO:teuthology.orchestra.run.trial198.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-21T14:39:26.741 INFO:teuthology.orchestra.run.trial198.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-21T14:39:26.754 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-21T14:39:26.757 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-21T14:39:26.763 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-21T14:39:26.765 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-21T14:39:26.776 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-21T14:39:26.782 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-21T14:39:26.806 INFO:teuthology.orchestra.run.trial198.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-21T14:39:26.810 INFO:teuthology.orchestra.run.trial198.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-21T14:39:26.860 INFO:teuthology.orchestra.run.trial198.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-21T14:39:26.872 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-21T14:39:26.875 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-21T14:39:26.879 INFO:teuthology.orchestra.run.trial198.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-21T14:39:26.885 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-21T14:39:26.889 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-21T14:39:26.892 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-21T14:39:26.907 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-21T14:39:26.911 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-21T14:39:26.916 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-21T14:39:26.927 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-21T14:39:26.936 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-21T14:39:26.943 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-21T14:39:26.962 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-21T14:39:26.973 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-21T14:39:26.989 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-21T14:39:26.993 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-21T14:39:27.001 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-21T14:39:27.041 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-21T14:39:27.365 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-21T14:39:27.375 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-21T14:39:27.378 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-21T14:39:27.384 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-21T14:39:27.386 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-21T14:39:27.414 INFO:teuthology.orchestra.run.trial198.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-21T14:39:27.461 INFO:teuthology.orchestra.run.trial198.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-21T14:39:27.471 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-21T14:39:27.477 INFO:teuthology.orchestra.run.trial198.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-21T14:39:27.480 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-21T14:39:27.486 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-21T14:39:27.490 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-21T14:39:27.497 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-21T14:39:27.500 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-21T14:39:27.528 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-21T14:39:27.537 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-21T14:39:27.543 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-21T14:39:27.550 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-21T14:39:27.584 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-21T14:39:27.810 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-21T14:39:27.834 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-21T14:39:27.837 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-21T14:39:27.896 INFO:teuthology.orchestra.run.trial198.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-21T14:39:27.898 INFO:teuthology.orchestra.run.trial198.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-21T14:39:27.919 INFO:teuthology.orchestra.run.trial198.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-21T14:39:28.243 INFO:teuthology.orchestra.run.trial198.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-21T14:39:28.316 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-21T14:39:28.989 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-21T14:39:29.005 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-21T14:39:29.015 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-21T14:39:29.022 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-21T14:39:29.035 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-21T14:39:29.051 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-21T14:39:29.126 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-21T14:39:29.135 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-21T14:39:29.158 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-21T14:39:29.188 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-21T14:39:29.238 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-21T14:39:29.244 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-21T14:39:29.247 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-21T14:39:29.249 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-21T14:39:29.266 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-21T14:39:29.266 INFO:teuthology.orchestra.run.trial198.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-21T14:39:29.266 INFO:teuthology.orchestra.run.trial198.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-21T14:39:29.266 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:29.273 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-21T14:39:29.296 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-21T14:39:29.296 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-21T14:39:29.296 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:29.330 INFO:teuthology.orchestra.run.trial198.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-21T14:39:29.920 INFO:teuthology.orchestra.run.trial198.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-21T14:39:35.280 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-classic-2:20.3.0-5076.g57019f91.el9.x86_6 80/96 2026-02-21T14:39:35.343 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-crimson-2:20.3.0-5076.g57019f91.el9.x86_6 81/96 2026-02-21T14:39:35.345 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-volume-2:20.3.0-5076.g57019f91.el9.noarch 82/96 2026-02-21T14:39:35.359 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-volume-2:20.3.0-5076.g57019f91.el9.noarch 82/96 2026-02-21T14:39:35.359 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:39:35.359 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-21T14:39:35.359 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:35.371 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-2:20.3.0-5076.g57019f91.el9.x86_64 83/96 2026-02-21T14:39:35.394 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-2:20.3.0-5076.g57019f91.el9.x86_64 83/96 2026-02-21T14:39:35.394 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:39:35.395 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-21T14:39:35.395 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-21T14:39:35.395 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-21T14:39:35.395 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:35.449 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: cephadm-2:20.3.0-5076.g57019f91.el9.noarch 84/96 2026-02-21T14:39:35.451 INFO:teuthology.orchestra.run.trial198.stdout: Installing : cephadm-2:20.3.0-5076.g57019f91.el9.noarch 84/96 2026-02-21T14:39:35.460 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5076.g57019f91.el9 85/96 2026-02-21T14:39:35.483 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5076.g57019f91.el 86/96 2026-02-21T14:39:35.485 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5076.g57019f91.el9.noarc 87/96 2026-02-21T14:39:37.053 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5076.g57019f91.el9.noarc 87/96 2026-02-21T14:39:37.060 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5076.g57019f91.el9.noa 88/96 2026-02-21T14:39:37.515 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5076.g57019f91.el9.noa 88/96 2026-02-21T14:39:37.516 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5076.g57019 89/96 2026-02-21T14:39:37.528 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5076.g57019 89/96 2026-02-21T14:39:37.528 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5076.g57019f91.el9.noa 90/96 2026-02-21T14:39:37.584 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5076.g57019f91.el9.noa 90/96 2026-02-21T14:39:37.623 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5076.g57019f91.el9. 91/96 2026-02-21T14:39:37.625 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-2:20.3.0-5076.g57019f91.el9.x86_64 92/96 2026-02-21T14:39:37.645 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5076.g57019f91.el9.x86_64 92/96 2026-02-21T14:39:37.645 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:39:37.645 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-21T14:39:37.645 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-21T14:39:37.645 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-21T14:39:37.645 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:37.656 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mgr-rook-2:20.3.0-5076.g57019f91.el9.noarch 93/96 2026-02-21T14:39:37.874 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5076.g57019f91.el9.noarch 93/96 2026-02-21T14:39:37.875 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mon-2:20.3.0-5076.g57019f91.el9.x86_64 94/96 2026-02-21T14:39:37.896 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mon-2:20.3.0-5076.g57019f91.el9.x86_64 94/96 2026-02-21T14:39:37.896 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:39:37.896 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-21T14:39:37.896 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-21T14:39:37.896 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-21T14:39:37.897 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:37.980 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-mds-2:20.3.0-5076.g57019f91.el9.x86_64 95/96 2026-02-21T14:39:38.000 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-mds-2:20.3.0-5076.g57019f91.el9.x86_64 95/96 2026-02-21T14:39:38.000 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:39:38.000 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-21T14:39:38.000 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-21T14:39:38.000 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-21T14:39:38.000 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:38.001 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-2:20.3.0-5076.g57019f91.el9.x86_64 96/96 2026-02-21T14:39:38.006 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5076.g57019f91.el9.x86_6 96/96 2026-02-21T14:39:38.029 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5076.g57019f91.el9.x86_6 96/96 2026-02-21T14:39:39.389 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-2:20.3.0-5076.g57019f91.el9.x86_64 96/96 2026-02-21T14:39:39.389 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-2:20.3.0-5076.g57019f91.el9.x86_64 1/96 2026-02-21T14:39:39.389 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mds-2:20.3.0-5076.g57019f91.el9.x86_64 2/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-2:20.3.0-5076.g57019f91.el9.x86_64 3/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mon-2:20.3.0-5076.g57019f91.el9.x86_64 4/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-2:20.3.0-5076.g57019f91.el9.x86_64 5/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-classic-2:20.3.0-5076.g57019f91.el9.x86_6 6/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5076.g57019f91.el9.x86_6 7/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5076.g57019f91.el 8/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5076.g57019f91.el9.noarc 9/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5076.g57019f91.el9.noa 10/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5076.g57019 11/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5076.g57019f91.el9.noa 12/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5076.g57019f91.el9. 13/96 2026-02-21T14:39:39.390 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5076.g57019f91.el9.noarch 14/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5076.g57019f91.el9 15/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-volume-2:20.3.0-5076.g57019f91.el9.noarch 16/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : cephadm-2:20.3.0-5076.g57019f91.el9.noarch 17/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-21T14:39:39.391 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-21T14:39:39.392 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-21T14:39:39.393 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-21T14:39:39.394 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-21T14:39:39.395 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-21T14:39:39.396 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-21T14:39:39.397 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-21T14:39:39.397 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-21T14:39:39.397 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-21T14:39:39.397 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-21T14:39:39.397 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-21T14:39:39.452 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-grafana-dashboards-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mds-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-cephadm-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-dashboard-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.453 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-k8sevents-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-modules-core-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mgr-rook-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-mon-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-classic-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-prometheus-alerts-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: ceph-volume-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: cephadm-2:20.3.0-5076.g57019f91.el9.noarch 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-21T14:39:39.454 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: lua-5.4.4-4.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-21T14:39:39.455 INFO:teuthology.orchestra.run.trial198.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-21T14:39:39.456 INFO:teuthology.orchestra.run.trial198.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-21T14:39:39.457 INFO:teuthology.orchestra.run.trial198.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-ply-3.11-14.el9.noarch 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-21T14:39:39.458 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-21T14:39:39.459 INFO:teuthology.orchestra.run.trial198.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: unzip-6.0-59.el9.x86_64 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-21T14:39:39.460 INFO:teuthology.orchestra.run.trial198.stdout: zip-3.0-35.el9.x86_64 2026-02-21T14:39:39.461 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:39.461 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:39.517 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-base 2026-02-21T14:39:39.831 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:35 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:39.896 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-base-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:39.914 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:39.915 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:39.915 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:39.936 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install cephadm 2026-02-21T14:39:40.247 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:36 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:40.312 INFO:teuthology.orchestra.run.trial198.stdout:Package cephadm-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:39:40.330 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:40.331 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:40.331 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:40.353 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-immutable-object-cache 2026-02-21T14:39:40.658 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:36 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:40.741 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:40.742 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:40.742 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-21T14:39:40.742 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:40.742 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:39:40.742 INFO:teuthology.orchestra.run.trial198.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5076.g57019f91.el9 ceph 149 k 2026-02-21T14:39:40.742 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:40.742 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:39:40.743 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:40.743 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-21T14:39:40.743 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:40.743 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 149 k 2026-02-21T14:39:40.743 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 455 k 2026-02-21T14:39:40.743 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:39:41.386 INFO:teuthology.orchestra.run.trial198.stdout:ceph-immutable-object-cache-20.3.0-5076.g57019f 232 kB/s | 149 kB 00:00 2026-02-21T14:39:41.386 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:39:41.387 INFO:teuthology.orchestra.run.trial198.stdout:Total 231 kB/s | 149 kB 00:00 2026-02-21T14:39:41.387 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:39:41.391 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:39:41.391 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:39:41.423 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:39:41.423 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:39:41.479 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:39:41.480 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5076.g57019f91. 1/1 2026-02-21T14:39:41.503 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5076.g57019f91. 1/1 2026-02-21T14:39:41.503 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:39:41.503 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-21T14:39:41.503 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:41.793 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5076.g57019f91. 1/1 2026-02-21T14:39:41.793 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:41.793 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:39:41.793 INFO:teuthology.orchestra.run.trial198.stdout: ceph-immutable-object-cache-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:41.793 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:41.794 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:41.821 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr 2026-02-21T14:39:42.135 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:38 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:42.200 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:42.218 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:42.219 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:42.219 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:42.242 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-dashboard 2026-02-21T14:39:42.546 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:38 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:42.610 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-dashboard-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:39:42.629 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:42.629 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:42.629 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:42.650 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-21T14:39:42.960 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:38 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:43.025 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:39:43.043 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:43.044 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:43.044 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:43.066 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-rook 2026-02-21T14:39:43.374 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:39 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:43.438 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-rook-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:39:43.457 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:43.458 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:43.458 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:43.479 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-cephadm 2026-02-21T14:39:43.789 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:39 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:43.854 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-cephadm-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:39:43.873 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:43.873 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:43.873 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:43.895 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd 2026-02-21T14:39:44.208 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:40 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:44.272 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:44.291 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:44.291 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:44.292 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:44.312 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-classic 2026-02-21T14:39:44.622 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:40 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:44.688 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-classic-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:44.707 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:44.707 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:44.707 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:44.728 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-fuse 2026-02-21T14:39:45.037 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:41 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:45.120 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:45.121 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:45.121 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-21T14:39:45.121 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout: ceph-fuse x86_64 2:20.3.0-5076.g57019f91.el9 ceph 914 k 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 914 k 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 2.9 M 2026-02-21T14:39:45.122 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:39:46.123 INFO:teuthology.orchestra.run.trial198.stdout:ceph-fuse-20.3.0-5076.g57019f91.el9.x86_64.rpm 913 kB/s | 914 kB 00:01 2026-02-21T14:39:46.124 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:39:46.124 INFO:teuthology.orchestra.run.trial198.stdout:Total 912 kB/s | 914 kB 00:01 2026-02-21T14:39:46.124 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:39:46.129 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:39:46.129 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:39:46.164 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:39:46.164 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:39:46.255 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:39:46.272 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:46.522 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:46.580 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:46.581 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:46.581 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:39:46.581 INFO:teuthology.orchestra.run.trial198.stdout: ceph-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:46.581 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:46.581 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:46.609 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-volume 2026-02-21T14:39:46.919 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:42 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:46.984 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-volume-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:39:47.003 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:47.003 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:47.003 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:47.025 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados-devel 2026-02-21T14:39:47.338 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:43 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:47.422 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:47.422 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout: librados-devel x86_64 2:20.3.0-5076.g57019f91.el9 ceph 124 k 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:47.423 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 124 k 2026-02-21T14:39:47.424 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 434 k 2026-02-21T14:39:47.424 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:39:47.920 INFO:teuthology.orchestra.run.trial198.stdout:librados-devel-20.3.0-5076.g57019f91.el9.x86_64 249 kB/s | 124 kB 00:00 2026-02-21T14:39:47.921 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:39:47.921 INFO:teuthology.orchestra.run.trial198.stdout:Total 248 kB/s | 124 kB 00:00 2026-02-21T14:39:47.921 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:39:47.923 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:39:47.923 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:39:47.955 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:39:47.956 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:39:48.010 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:39:48.027 INFO:teuthology.orchestra.run.trial198.stdout: Installing : librados-devel-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:48.109 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: librados-devel-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:48.169 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : librados-devel-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:48.170 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:48.170 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:39:48.170 INFO:teuthology.orchestra.run.trial198.stdout: librados-devel-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:48.170 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:48.170 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:48.200 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs2 2026-02-21T14:39:48.510 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:44 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:48.574 INFO:teuthology.orchestra.run.trial198.stdout:Package libcephfs2-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:48.593 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:48.594 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:48.594 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:48.615 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs-devel 2026-02-21T14:39:48.924 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:44 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:49.007 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-devel x86_64 2:20.3.0-5076.g57019f91.el9 ceph 37 k 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:39:49.008 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:49.009 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-21T14:39:49.009 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:49.009 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 37 k 2026-02-21T14:39:49.009 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 171 k 2026-02-21T14:39:49.009 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:39:49.368 INFO:teuthology.orchestra.run.trial198.stdout:libcephfs-devel-20.3.0-5076.g57019f91.el9.x86_6 103 kB/s | 37 kB 00:00 2026-02-21T14:39:49.369 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:39:49.369 INFO:teuthology.orchestra.run.trial198.stdout:Total 103 kB/s | 37 kB 00:00 2026-02-21T14:39:49.369 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:39:49.372 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:39:49.372 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:39:49.375 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:39:49.375 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:39:49.396 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:39:49.412 INFO:teuthology.orchestra.run.trial198.stdout: Installing : libcephfs-devel-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:49.470 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:49.528 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : libcephfs-devel-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:49.528 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:49.528 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:39:49.528 INFO:teuthology.orchestra.run.trial198.stdout: libcephfs-devel-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:49.528 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:49.528 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:49.555 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados2 2026-02-21T14:39:49.869 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:45 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:49.935 INFO:teuthology.orchestra.run.trial198.stdout:Package librados2-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:49.954 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:49.954 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:49.955 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:49.977 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librbd1 2026-02-21T14:39:50.285 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:46 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:50.349 INFO:teuthology.orchestra.run.trial198.stdout:Package librbd1-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:50.368 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:50.369 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:50.369 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:50.390 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rados 2026-02-21T14:39:50.698 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:46 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:50.762 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rados-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:50.781 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:50.781 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:50.781 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:50.803 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rgw 2026-02-21T14:39:51.112 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:47 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:51.176 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rgw-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:51.195 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:51.196 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:51.196 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:51.217 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-cephfs 2026-02-21T14:39:51.531 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:47 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:51.596 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-cephfs-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:51.615 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:51.615 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:51.615 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:51.637 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rbd 2026-02-21T14:39:51.950 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:47 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:52.016 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rbd-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:39:52.034 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:52.035 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:39:52.035 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:52.056 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-fuse 2026-02-21T14:39:52.368 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:48 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:52.452 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:52.452 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout: rbd-fuse x86_64 2:20.3.0-5076.g57019f91.el9 ceph 86 k 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:52.453 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 86 k 2026-02-21T14:39:52.454 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 236 k 2026-02-21T14:39:52.454 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:39:52.879 INFO:teuthology.orchestra.run.trial198.stdout:rbd-fuse-20.3.0-5076.g57019f91.el9.x86_64.rpm 203 kB/s | 86 kB 00:00 2026-02-21T14:39:52.880 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:39:52.880 INFO:teuthology.orchestra.run.trial198.stdout:Total 202 kB/s | 86 kB 00:00 2026-02-21T14:39:52.880 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:39:52.885 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:39:52.885 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:39:52.917 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:39:52.917 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:39:52.969 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:39:52.986 INFO:teuthology.orchestra.run.trial198.stdout: Installing : rbd-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:53.065 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:53.124 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : rbd-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:39:53.124 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:53.124 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:39:53.124 INFO:teuthology.orchestra.run.trial198.stdout: rbd-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:39:53.124 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:53.124 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:39:53.154 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-mirror 2026-02-21T14:39:53.464 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:49 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:39:53.547 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:39:53.548 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:53.548 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-21T14:39:53.548 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:53.548 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:39:53.548 INFO:teuthology.orchestra.run.trial198.stdout: rbd-mirror x86_64 2:20.3.0-5076.g57019f91.el9 ceph 2.6 M 2026-02-21T14:39:53.548 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:53.549 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:39:53.549 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:39:53.549 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-21T14:39:53.549 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:39:53.549 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 2.6 M 2026-02-21T14:39:53.549 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 11 M 2026-02-21T14:39:53.549 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:40:00.593 INFO:teuthology.orchestra.run.trial198.stdout:rbd-mirror-20.3.0-5076.g57019f91.el9.x86_64.rpm 377 kB/s | 2.6 MB 00:07 2026-02-21T14:40:00.593 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:40:00.594 INFO:teuthology.orchestra.run.trial198.stdout:Total 377 kB/s | 2.6 MB 00:07 2026-02-21T14:40:00.594 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:40:00.599 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:40:00.599 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:40:00.639 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:40:00.639 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:40:00.810 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:40:00.812 INFO:teuthology.orchestra.run.trial198.stdout: Installing : rbd-mirror-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:40:00.846 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:40:00.846 INFO:teuthology.orchestra.run.trial198.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T14:40:00.846 INFO:teuthology.orchestra.run.trial198.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-21T14:40:00.846 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-21T14:40:00.847 INFO:teuthology.orchestra.run.trial198.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-21T14:40:00.847 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:01.147 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : rbd-mirror-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:40:01.147 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:01.147 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:40:01.147 INFO:teuthology.orchestra.run.trial198.stdout: rbd-mirror-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:40:01.148 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:01.148 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:40:01.176 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-nbd 2026-02-21T14:40:01.494 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:57 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:40:01.577 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:40:01.578 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:40:01.578 INFO:teuthology.orchestra.run.trial198.stdout: Package Architecture Version Repository Size 2026-02-21T14:40:01.578 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:40:01.578 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout: rbd-nbd x86_64 2:20.3.0-5076.g57019f91.el9 ceph 166 k 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout:Install 1 Package 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 166 k 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 496 k 2026-02-21T14:40:01.579 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:40:02.452 INFO:teuthology.orchestra.run.trial198.stdout:rbd-nbd-20.3.0-5076.g57019f91.el9.x86_64.rpm 190 kB/s | 166 kB 00:00 2026-02-21T14:40:02.453 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:40:02.453 INFO:teuthology.orchestra.run.trial198.stdout:Total 190 kB/s | 166 kB 00:00 2026-02-21T14:40:02.453 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:40:02.458 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:40:02.458 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:40:02.490 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:40:02.490 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:40:02.546 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:40:02.563 INFO:teuthology.orchestra.run.trial198.stdout: Installing : rbd-nbd-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:40:02.649 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:40:02.709 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : rbd-nbd-2:20.3.0-5076.g57019f91.el9.x86_64 1/1 2026-02-21T14:40:02.709 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:02.709 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:40:02.710 INFO:teuthology.orchestra.run.trial198.stdout: rbd-nbd-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:40:02.710 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:02.710 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:40:02.740 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-crimson 2026-02-21T14:40:03.050 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:59 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:40:03.116 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-crimson-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:40:03.134 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:40:03.135 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:40:03.135 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:40:03.158 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-21T14:40:03.467 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:09:59 ago on Sat 21 Feb 2026 02:30:04 PM UTC. 2026-02-21T14:40:03.552 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:40:03.553 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:40:03.553 INFO:teuthology.orchestra.run.trial198.stdout: Package Arch Version Repo Size 2026-02-21T14:40:03.553 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:40:03.553 INFO:teuthology.orchestra.run.trial198.stdout:Installing: 2026-02-21T14:40:03.553 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5076.g57019f91.el9 ceph 429 M 2026-02-21T14:40:03.553 INFO:teuthology.orchestra.run.trial198.stdout:Installing weak dependencies: 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout: ceph-debugsource x86_64 2:20.3.0-5076.g57019f91.el9 ceph 12 M 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout:Transaction Summary 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout:================================================================================ 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout:Install 2 Packages 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout:Total download size: 441 M 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout:Installed size: 2.7 G 2026-02-21T14:40:03.554 INFO:teuthology.orchestra.run.trial198.stdout:Downloading Packages: 2026-02-21T14:40:38.199 INFO:teuthology.orchestra.run.trial198.stdout:(1/2): ceph-debugsource-20.3.0-5076.g57019f91.e 355 kB/s | 12 MB 00:34 2026-02-21T14:52:23.349 INFO:teuthology.orchestra.run.trial198.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5076.g 593 kB/s | 429 MB 12:19 2026-02-21T14:52:23.350 INFO:teuthology.orchestra.run.trial198.stdout:-------------------------------------------------------------------------------- 2026-02-21T14:52:23.350 INFO:teuthology.orchestra.run.trial198.stdout:Total 610 kB/s | 441 MB 12:19 2026-02-21T14:52:23.351 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction check 2026-02-21T14:52:23.356 INFO:teuthology.orchestra.run.trial198.stdout:Transaction check succeeded. 2026-02-21T14:52:23.356 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction test 2026-02-21T14:52:24.722 INFO:teuthology.orchestra.run.trial198.stdout:Transaction test succeeded. 2026-02-21T14:52:24.723 INFO:teuthology.orchestra.run.trial198.stdout:Running transaction 2026-02-21T14:52:28.307 INFO:teuthology.orchestra.run.trial198.stdout: Preparing : 1/1 2026-02-21T14:52:51.409 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-debugsource-2:20.3.0-5076.g57019f91.el9.x86_64 1/2 2026-02-21T14:52:51.426 INFO:teuthology.orchestra.run.trial198.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5076.g57019f91.e 2/2 2026-02-21T14:52:52.272 INFO:teuthology.orchestra.run.trial198.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5076.g57019f91.e 2/2 2026-02-21T14:52:52.272 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-debugsource-2:20.3.0-5076.g57019f91.el9.x86_64 1/2 2026-02-21T14:52:52.331 INFO:teuthology.orchestra.run.trial198.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5076.g57019f91.e 2/2 2026-02-21T14:52:52.331 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:52:52.331 INFO:teuthology.orchestra.run.trial198.stdout:Installed: 2026-02-21T14:52:52.331 INFO:teuthology.orchestra.run.trial198.stdout: ceph-debugsource-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:52:52.331 INFO:teuthology.orchestra.run.trial198.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5076.g57019f91.el9.x86_64 2026-02-21T14:52:52.331 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:52:52.331 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:52:52.404 DEBUG:teuthology.parallel:result is None 2026-02-21T14:52:52.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=57019f91dea307b3c704ce2392a5f854512b5900 2026-02-21T14:52:52.571 DEBUG:teuthology.orchestra.run.trial198:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-21T14:52:52.594 INFO:teuthology.orchestra.run.trial198.stdout:20.3.0-5076.g57019f91.el9 2026-02-21T14:52:52.594 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5076.g57019f91.el9 2026-02-21T14:52:52.595 INFO:teuthology.task.install:The correct ceph version 20.3.0-5076.g57019f91 is installed. 2026-02-21T14:52:52.597 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-21T14:52:52.597 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:52:52.597 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-21T14:52:52.660 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-21T14:52:52.661 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:52:52.661 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/usr/bin/daemon-helper 2026-02-21T14:52:52.724 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-21T14:52:52.788 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-21T14:52:52.789 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:52:52.789 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-21T14:52:52.853 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-21T14:52:52.917 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-21T14:52:52.918 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:52:52.918 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/usr/bin/stdin-killer 2026-02-21T14:52:52.982 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-21T14:52:53.046 INFO:teuthology.run_tasks:Running task pexec... 2026-02-21T14:52:53.063 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-21T14:52:53.064 DEBUG:teuthology.orchestra.run.trial198:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-21T14:52:53.088 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:52:53.088 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-21T14:52:53.139 DEBUG:teuthology.parallel:result is None 2026-02-21T14:52:53.139 INFO:teuthology.run_tasks:Running task install... 2026-02-21T14:52:53.153 DEBUG:teuthology.task.install:project ceph 2026-02-21T14:52:53.153 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900'}} 2026-02-21T14:52:53.153 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900'} 2026-02-21T14:52:53.153 INFO:teuthology.task.install:Using flavor: debug 2026-02-21T14:52:53.162 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-21T14:52:53.162 INFO:teuthology.task.install:extra packages: [] 2026-02-21T14:52:53.162 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900', 'tag': None, 'wait_for_package': False} 2026-02-21T14:52:53.163 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=57019f91dea307b3c704ce2392a5f854512b5900 2026-02-21T14:52:53.323 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/ 2026-02-21T14:52:53.323 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5076.g57019f91 2026-02-21T14:52:53.720 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-trans-cc-opt-rbm-drop-brf/57019f91dea307b3c704ce2392a5f854512b5900/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-21T14:52:53.720 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:52:53.720 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-21T14:52:53.746 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-21T14:52:53.746 DEBUG:teuthology.orchestra.run.trial198:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/57019f91dea307b3c704ce2392a5f854512b5900/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-21T14:52:53.812 DEBUG:teuthology.orchestra.run.trial198:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-21T14:52:53.875 DEBUG:teuthology.orchestra.run.trial198:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-21T14:52:53.933 INFO:teuthology.orchestra.run.trial198.stdout:check_obsoletes = 1 2026-02-21T14:52:53.944 DEBUG:teuthology.orchestra.run.trial198:> sudo yum clean all 2026-02-21T14:52:54.104 INFO:teuthology.orchestra.run.trial198.stdout:56 files removed 2026-02-21T14:52:54.120 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-radosgw 2026-02-21T14:52:55.281 INFO:teuthology.orchestra.run.trial198.stdout:ceph packages for x86_64 93 kB/s | 94 kB 00:01 2026-02-21T14:52:56.018 INFO:teuthology.orchestra.run.trial198.stdout:ceph noarch packages 34 kB/s | 24 kB 00:00 2026-02-21T14:52:56.614 INFO:teuthology.orchestra.run.trial198.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-21T14:53:05.294 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.9 MB 00:08 2026-02-21T14:53:32.448 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 27 MB 00:26 2026-02-21T14:53:36.605 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 7.8 MB 00:01 2026-02-21T14:53:37.447 INFO:teuthology.orchestra.run.trial198.stdout:CentOS Stream 9 - Extras packages 194 kB/s | 20 kB 00:00 2026-02-21T14:53:38.003 INFO:teuthology.orchestra.run.trial198.stdout:Extra Packages for Enterprise Linux 42 MB/s | 20 MB 00:00 2026-02-21T14:53:41.166 INFO:teuthology.orchestra.run.trial198.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-21T14:53:42.346 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-radosgw-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:42.365 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:42.366 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:42.366 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:42.390 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-test 2026-02-21T14:53:42.702 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:01 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:42.765 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-test-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:42.784 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:42.785 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:42.785 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:42.806 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph 2026-02-21T14:53:43.118 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:02 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:43.183 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:43.202 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:43.202 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:43.202 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:43.224 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-base 2026-02-21T14:53:43.530 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:02 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:43.594 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-base-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:43.612 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:43.613 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:43.613 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:43.634 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install cephadm 2026-02-21T14:53:43.946 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:02 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:44.010 INFO:teuthology.orchestra.run.trial198.stdout:Package cephadm-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:53:44.029 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:44.029 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:44.030 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:44.051 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-immutable-object-cache 2026-02-21T14:53:44.365 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:03 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:44.430 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-immutable-object-cache-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:44.449 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:44.450 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:44.450 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:44.472 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr 2026-02-21T14:53:44.780 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:03 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:44.845 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:44.864 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:44.864 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:44.865 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:44.886 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-dashboard 2026-02-21T14:53:45.195 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:04 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:45.261 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-dashboard-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:53:45.279 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:45.280 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:45.280 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:45.302 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-21T14:53:45.615 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:04 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:45.679 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:53:45.698 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:45.698 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:45.699 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:45.720 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-rook 2026-02-21T14:53:46.035 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:05 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:46.100 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-rook-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:53:46.119 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:46.120 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:46.120 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:46.143 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-mgr-cephadm 2026-02-21T14:53:46.459 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:05 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:46.526 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-mgr-cephadm-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:53:46.545 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:46.545 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:46.545 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:46.568 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd 2026-02-21T14:53:46.882 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:05 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:46.946 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:46.965 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:46.965 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:46.966 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:46.988 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-osd-classic 2026-02-21T14:53:47.297 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:06 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:47.362 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-osd-classic-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:47.381 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:47.381 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:47.382 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:47.404 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-fuse 2026-02-21T14:53:47.720 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:06 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:47.784 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:47.802 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:47.803 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:47.803 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:47.825 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install ceph-volume 2026-02-21T14:53:48.133 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:07 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:48.198 INFO:teuthology.orchestra.run.trial198.stdout:Package ceph-volume-2:20.3.0-5076.g57019f91.el9.noarch is already installed. 2026-02-21T14:53:48.216 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:48.217 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:48.217 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:48.239 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados-devel 2026-02-21T14:53:48.546 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:07 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:48.611 INFO:teuthology.orchestra.run.trial198.stdout:Package librados-devel-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:48.630 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:48.630 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:48.631 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:48.653 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs2 2026-02-21T14:53:48.968 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:07 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:49.033 INFO:teuthology.orchestra.run.trial198.stdout:Package libcephfs2-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:49.051 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:49.052 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:49.052 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:49.074 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install libcephfs-devel 2026-02-21T14:53:49.385 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:08 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:49.450 INFO:teuthology.orchestra.run.trial198.stdout:Package libcephfs-devel-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:49.468 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:49.469 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:49.469 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:49.491 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librados2 2026-02-21T14:53:49.800 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:08 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:49.864 INFO:teuthology.orchestra.run.trial198.stdout:Package librados2-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:49.883 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:49.883 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:49.883 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:49.905 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install librbd1 2026-02-21T14:53:50.222 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:09 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:50.286 INFO:teuthology.orchestra.run.trial198.stdout:Package librbd1-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:50.304 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:50.305 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:50.305 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:50.327 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rados 2026-02-21T14:53:50.637 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:09 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:50.702 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rados-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:50.720 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:50.721 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:50.721 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:50.743 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rgw 2026-02-21T14:53:51.055 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:10 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:51.120 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rgw-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:51.139 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:51.139 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:51.139 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:51.162 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-cephfs 2026-02-21T14:53:51.471 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:10 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:51.536 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-cephfs-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:51.555 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:51.556 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:51.556 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:51.579 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install python3-rbd 2026-02-21T14:53:51.886 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:10 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:51.951 INFO:teuthology.orchestra.run.trial198.stdout:Package python3-rbd-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:51.970 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:51.971 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:51.971 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:51.992 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-fuse 2026-02-21T14:53:52.312 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:11 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:52.376 INFO:teuthology.orchestra.run.trial198.stdout:Package rbd-fuse-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:52.394 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:52.395 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:52.395 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:52.418 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-mirror 2026-02-21T14:53:52.727 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:11 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:52.793 INFO:teuthology.orchestra.run.trial198.stdout:Package rbd-mirror-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:52.811 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:52.812 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:52.812 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:52.834 DEBUG:teuthology.orchestra.run.trial198:> sudo yum -y install rbd-nbd 2026-02-21T14:53:53.154 INFO:teuthology.orchestra.run.trial198.stdout:Last metadata expiration check: 0:00:12 ago on Sat 21 Feb 2026 02:53:41 PM UTC. 2026-02-21T14:53:53.218 INFO:teuthology.orchestra.run.trial198.stdout:Package rbd-nbd-2:20.3.0-5076.g57019f91.el9.x86_64 is already installed. 2026-02-21T14:53:53.237 INFO:teuthology.orchestra.run.trial198.stdout:Dependencies resolved. 2026-02-21T14:53:53.237 INFO:teuthology.orchestra.run.trial198.stdout:Nothing to do. 2026-02-21T14:53:53.238 INFO:teuthology.orchestra.run.trial198.stdout:Complete! 2026-02-21T14:53:53.263 DEBUG:teuthology.parallel:result is None 2026-02-21T14:53:53.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=57019f91dea307b3c704ce2392a5f854512b5900 2026-02-21T14:53:53.427 DEBUG:teuthology.orchestra.run.trial198:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-21T14:53:53.448 INFO:teuthology.orchestra.run.trial198.stdout:20.3.0-5076.g57019f91.el9 2026-02-21T14:53:53.448 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5076.g57019f91.el9 2026-02-21T14:53:53.448 INFO:teuthology.task.install:The correct ceph version 20.3.0-5076.g57019f91 is installed. 2026-02-21T14:53:53.450 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-21T14:53:53.450 INFO:teuthology.run_tasks:Running task ceph... 2026-02-21T14:53:53.555 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-21T14:53:53.555 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 777 /var/log/ceph 2026-02-21T14:53:53.579 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-21T14:53:53.580 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-21T14:53:53.642 INFO:tasks.ceph:Creating extra log directories... 2026-02-21T14:53:53.642 DEBUG:teuthology.orchestra.run.trial198:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-21T14:53:53.717 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-21T14:53:53.718 INFO:tasks.ceph:config {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 25, 'debug seastore backref': 20, 'debug seastore cache': 25, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 25, 'debug seastore odata': 25, 'debug seastore omap': 25, 'debug seastore onode': 25, 'debug seastore t': 25, 'debug seastore tm': 25, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-21T14:53:53.718 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697', 'branch': 'wip-trans-cc-opt-rbm-drop-brf', 'description': 'crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/rbm$/{crimson_seastore_rbm} rados}', 'email': 'xxhdx1985126@gmail.com', 'first_in_suite': False, 'flavor': 'debug', 'job_id': '62697', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-trans-cc-opt-rbm-drop-brf'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}, 'osd': {'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 25, 'debug seastore backref': 20, 'debug seastore cache': 25, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 25, 'debug seastore odata': 25, 'debug seastore omap': 25, 'debug seastore onode': 25, 'debug seastore t': 25, 'debug seastore tm': 25, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900'}}, 'workunit': {'branch': 'wip-seastore-qa-log-level-up', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f'}}, 'owner': 'scheduled_xuxuehan@soko04.front.sepia.ceph.com', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2477, 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-seastore-qa-log-level-up', 'suite_path': '/home/teuthworker/src/github.com_xxhdx1985126_ceph_c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/xxhdx1985126/ceph.git', 'suite_sha1': 'c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f', 'targets': {'trial198.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBhqKVVg8S9YFBk7J8qT75wRENCUl+fIgkPYS9SpXCxZlsytpav9rmVXW/5ICZ5bD/g3G79EDa9+3al5ZV8NWg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 25, 'debug seastore backref': 20, 'debug seastore cache': 25, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 25, 'debug seastore odata': 25, 'debug seastore omap': 25, 'debug seastore onode': 25, 'debug seastore t': 25, 'debug seastore tm': 25, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'crimson_compat': True, 'flavor': 'debug', 'sha1': '57019f91dea307b3c704ce2392a5f854512b5900', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cephtool/test_crimson_dump_metrics.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-21_14:23:14', 'tube': 'trial', 'user': 'xuxuehan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699'} 2026-02-21T14:53:53.718 DEBUG:teuthology.orchestra.run.trial198:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-21T14:53:53.765 DEBUG:teuthology.orchestra.run.trial198:> sudo install -d -m0777 -- /var/run/ceph 2026-02-21T14:53:53.829 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:53:53.829 DEBUG:teuthology.orchestra.run.trial198:> dd if=/scratch_devs of=/dev/stdout 2026-02-21T14:53:53.887 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-21T14:53:53.888 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_1 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-21 14:40:01.005280331 +0000 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:53.944 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:53.945 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-21T14:53:54.007 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-21T14:53:54.007 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-21T14:53:54.007 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.000151151 s, 3.4 MB/s 2026-02-21T14:53:54.009 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-21T14:53:54.068 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_2 2026-02-21T14:53:54.124 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 1287 Links: 1 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-21 14:40:01.006280309 +0000 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-21 14:30:11.248710044 +0000 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-21 14:30:11.248710044 +0000 2026-02-21T14:53:54.125 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-21 14:30:11.248710044 +0000 2026-02-21T14:53:54.126 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-21T14:53:54.188 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-21T14:53:54.188 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-21T14:53:54.188 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.0001456 s, 3.5 MB/s 2026-02-21T14:53:54.190 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-21T14:53:54.247 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_3 2026-02-21T14:53:54.303 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-21T14:53:54.303 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-21T14:53:54.303 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-21T14:53:54.304 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-21T14:53:54.304 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-21T14:53:54.304 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-21 14:40:01.006280309 +0000 2026-02-21T14:53:54.304 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:54.304 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:54.304 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:54.304 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-21T14:53:54.367 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-21T14:53:54.367 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-21T14:53:54.367 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.000160618 s, 3.2 MB/s 2026-02-21T14:53:54.368 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-21T14:53:54.426 DEBUG:teuthology.orchestra.run.trial198:> stat /dev/vg_nvme/lv_4 2026-02-21T14:53:54.483 INFO:teuthology.orchestra.run.trial198.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-21T14:53:54.483 INFO:teuthology.orchestra.run.trial198.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-21T14:53:54.484 INFO:teuthology.orchestra.run.trial198.stdout:Device: 6h/6d Inode: 1291 Links: 1 2026-02-21T14:53:54.484 INFO:teuthology.orchestra.run.trial198.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-21T14:53:54.484 INFO:teuthology.orchestra.run.trial198.stdout:Context: system_u:object_r:device_t:s0 2026-02-21T14:53:54.484 INFO:teuthology.orchestra.run.trial198.stdout:Access: 2026-02-21 14:40:01.006280309 +0000 2026-02-21T14:53:54.484 INFO:teuthology.orchestra.run.trial198.stdout:Modify: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:54.484 INFO:teuthology.orchestra.run.trial198.stdout:Change: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:54.484 INFO:teuthology.orchestra.run.trial198.stdout: Birth: 2026-02-21 14:30:11.249710007 +0000 2026-02-21T14:53:54.484 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-21T14:53:54.546 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records in 2026-02-21T14:53:54.547 INFO:teuthology.orchestra.run.trial198.stderr:1+0 records out 2026-02-21T14:53:54.547 INFO:teuthology.orchestra.run.trial198.stderr:512 bytes copied, 0.000137435 s, 3.7 MB/s 2026-02-21T14:53:54.548 DEBUG:teuthology.orchestra.run.trial198:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-21T14:53:54.605 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-21T14:53:54.605 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial198.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-21T14:53:54.605 INFO:tasks.ceph:Generating config... 2026-02-21T14:53:54.608 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] crimson cpu num = 2 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] debug seastore = 25 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-21T14:53:54.608 INFO:tasks.ceph:[osd] debug seastore cache = 25 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore lba = 25 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore odata = 25 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore omap = 25 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore onode = 25 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore t = 25 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] debug seastore tm = 25 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-21T14:53:54.609 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-21T14:53:54.610 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-21T14:53:54.610 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-21T14:53:54.610 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-21T14:53:54.610 INFO:tasks.ceph:[mon] debug mon = 30 2026-02-21T14:53:54.610 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-21T14:53:54.611 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-21T14:53:54.611 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-21T14:53:54.611 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-21T14:53:54.611 INFO:tasks.ceph:Setting up mon.a... 2026-02-21T14:53:54.611 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-21T14:53:54.685 INFO:teuthology.orchestra.run.trial198.stdout:creating /etc/ceph/ceph.keyring 2026-02-21T14:53:54.688 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-21T14:53:54.773 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-21T14:53:54.798 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.198'), ('mon.b', '[v2:10.20.193.198:3301,v1:10.20.193.198:6790]'), ('mon.c', '[v2:10.20.193.198:3302,v1:10.20.193.198:6791]')] 2026-02-21T14:53:54.799 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.198,[v2:10.20.193.198:3301,v1:10.20.193.198:6790],[v2:10.20.193.198:3302,v1:10.20.193.198:6791]', 'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 25, 'debug seastore backref': 20, 'debug seastore cache': 25, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 25, 'debug seastore odata': 25, 'debug seastore omap': 25, 'debug seastore onode': 25, 'debug seastore t': 25, 'debug seastore tm': 25, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 30, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-21T14:53:54.800 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:53:54.800 DEBUG:teuthology.orchestra.run.trial198:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-21T14:53:54.855 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.198 --addv b '[v2:10.20.193.198:3301,v1:10.20.193.198:6790]' --addv c '[v2:10.20.193.198:3302,v1:10.20.193.198:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-21T14:53:54.938 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:53:54.947+0000 7fd5c6cfdf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:53:54.938 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:53:54.947+0000 7fd5c6cfdf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:53:54.938 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:53:54.947+0000 7fd5c6cfdf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:53:54.939 INFO:teuthology.orchestra.run.trial198.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:monmaptool: generated fsid 7ba0c362-e064-4d8f-9e0d-89e7be54cc27 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:setting min_mon_release = tentacle 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:epoch 0 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:fsid 7ba0c362-e064-4d8f-9e0d-89e7be54cc27 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:last_changed 2026-02-21T14:53:54.949453+0000 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:created 2026-02-21T14:53:54.949453+0000 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:min_mon_release 20 (tentacle) 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:election_strategy: 1 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:0: [v2:10.20.193.198:3300/0,v1:10.20.193.198:6789/0] mon.a 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:1: [v2:10.20.193.198:3301/0,v1:10.20.193.198:6790/0] mon.b 2026-02-21T14:53:54.940 INFO:teuthology.orchestra.run.trial198.stdout:2: [v2:10.20.193.198:3302/0,v1:10.20.193.198:6791/0] mon.c 2026-02-21T14:53:54.941 INFO:teuthology.orchestra.run.trial198.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-21T14:53:54.941 DEBUG:teuthology.orchestra.run.trial198:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-21T14:53:54.996 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7ba0c362-e064-4d8f-9e0d-89e7be54cc27... 2026-02-21T14:53:54.997 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-21T14:53:55.078 INFO:teuthology.orchestra.run.trial198.stdout:[global] 2026-02-21T14:53:55.078 INFO:teuthology.orchestra.run.trial198.stdout: chdir = "" 2026-02-21T14:53:55.078 INFO:teuthology.orchestra.run.trial198.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-21T14:53:55.078 INFO:teuthology.orchestra.run.trial198.stdout: auth supported = cephx 2026-02-21T14:53:55.078 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.078 INFO:teuthology.orchestra.run.trial198.stdout: filestore xattr use omap = true 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: mon clock drift allowed = 1.000 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: osd crush chooseleaf type = 0 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: auth debug = true 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: ms die on old message = true 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: ms die on bug = true 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-21T14:53:55.079 INFO:teuthology.orchestra.run.trial198.stdout: mon pg warn max object skew = 0 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: # disable pg_autoscaler by default for new pools 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: osd pool default size = 2 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: mon osd allow primary affinity = true 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: mon osd allow pg remap = true 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on legacy crush tunables = false 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on crush straw calc version zero = false 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on no sortbitwise = false 2026-02-21T14:53:55.080 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on osd down out interval zero = false 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: mon warn on too few osds = false 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: mon_allow_pool_size_one = true 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: osd default data pool replay window = 5 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: mon allow pool delete = true 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.081 INFO:teuthology.orchestra.run.trial198.stdout: mon cluster log file level = debug 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: debug asserts on shutdown = true 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: mon health detail to clog = false 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: mon host = "10.20.193.198,[v2:10.20.193.198:3301,v1:10.20.193.198:6790],[v2:10.20.193.198:3302,v1:10.20.193.198:6791]" 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: fsid = 7ba0c362-e064-4d8f-9e0d-89e7be54cc27 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout:[osd] 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: osd journal size = 100 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: osd scrub load threshold = 5.0 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: osd scrub max interval = 600 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: osd mclock profile = high_recovery_ops 2026-02-21T14:53:55.082 INFO:teuthology.orchestra.run.trial198.stdout: osd mclock skip benchmark = True 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd recover clone overlap = true 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd recovery max chunk = 1048576 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd debug shutdown = true 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd debug op order = true 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd debug verify stray on activate = true 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd debug trim objects = true 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd open classes on start = true 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd debug pg log writeout = true 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.083 INFO:teuthology.orchestra.run.trial198.stdout: osd deep scrub update digest min age = 30 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: osd map max advance = 10 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: journal zero on create = true 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: filestore ondisk finisher threads = 3 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: filestore apply finisher threads = 3 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: bdev debug aio = true 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: osd debug misdirected ops = true 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: crimson cpu num = 2 2026-02-21T14:53:55.084 INFO:teuthology.orchestra.run.trial198.stdout: osd min pg log entries = 5 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: bluestore zero block detection = True 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: crimson osd obc lru size = 10 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug ms = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug osd = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore = 25 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore backref = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore cache = 25 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore cleaner = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore device = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore epm = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore fixedkv tree = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore journal = 20 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore lba = 25 2026-02-21T14:53:55.085 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore odata = 25 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore omap = 25 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore onode = 25 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore t = 25 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: debug seastore tm = 25 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: osd debug verify cached snaps = True 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: osd debug verify missing on start = True 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: osd mclock override recovery settings = True 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: osd objectstore = seastore 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: osd op queue = debug_random 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: osd op queue cut off = debug_random 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: seastore cachepin size pershard = 64M 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-21T14:53:55.086 INFO:teuthology.orchestra.run.trial198.stdout: seastore max concurrent transactions = 8 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout:[mgr] 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: debug ms = 1 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: debug mgr = 20 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: debug mon = 20 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: debug auth = 20 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min pgs per osd = 4 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min bytes per osd = 10 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: mgr/telemetry/nag = false 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout:[mon] 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: debug ms = 1 2026-02-21T14:53:55.087 INFO:teuthology.orchestra.run.trial198.stdout: debug mon = 30 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: debug paxos = 20 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: debug auth = 20 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: mon data avail warn = 5 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: mon mgr mkfs grace = 240 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min pgs per osd = 4 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: mon osd reporter subtree level = osd 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: mon osd prime pg temp = true 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: mon reweight min bytes per osd = 10 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: auth mon ticket ttl = 660 # 11m 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: auth service ticket ttl = 240 # 4m 2026-02-21T14:53:55.088 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: # don't complain about insecure global_id in the test suite 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: # 1m isn't quite enough 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: mon_down_mkfs_grace = 2m 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: mon_warn_on_filestore_osds = false 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: mon scrub interval = 300 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: osd pool default crimson = True 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout:[client] 2026-02-21T14:53:55.089 INFO:teuthology.orchestra.run.trial198.stdout: rgw cache enabled = true 2026-02-21T14:53:55.090 INFO:teuthology.orchestra.run.trial198.stdout: rgw enable ops log = true 2026-02-21T14:53:55.090 INFO:teuthology.orchestra.run.trial198.stdout: rgw enable usage log = true 2026-02-21T14:53:55.090 INFO:teuthology.orchestra.run.trial198.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-21T14:53:55.090 INFO:teuthology.orchestra.run.trial198.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-21T14:53:55.090 INFO:teuthology.orchestra.run.trial198.stdout:[mon.a] 2026-02-21T14:53:55.090 INFO:teuthology.orchestra.run.trial198.stdout:[mon.b] 2026-02-21T14:53:55.090 INFO:teuthology.orchestra.run.trial198.stdout:[mon.c] 2026-02-21T14:53:55.091 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-21T14:53:55.091 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-21T14:53:55.181 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-21T14:53:55.181 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:53:55.181 DEBUG:teuthology.orchestra.run.trial198:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-21T14:53:55.197 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:53:55.197 DEBUG:teuthology.orchestra.run.trial198:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-21T14:53:55.255 INFO:tasks.ceph:Sending monmap to node ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:53:55.255 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:53:55.255 DEBUG:teuthology.orchestra.run.trial198:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-21T14:53:55.256 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-21T14:53:55.329 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:53:55.329 DEBUG:teuthology.orchestra.run.trial198:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-21T14:53:55.385 INFO:tasks.ceph:Setting up mon nodes... 2026-02-21T14:53:55.385 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-21T14:53:55.386 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-21T14:53:55.476 INFO:teuthology.orchestra.run.trial198.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-21T14:53:55.478 INFO:tasks.ceph:Setting up mds nodes... 2026-02-21T14:53:55.479 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-21T14:53:55.479 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-21T14:53:55.521 INFO:teuthology.orchestra.run.trial198.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-21T14:53:55.533 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-21T14:53:55.534 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial198.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-21T14:53:55.534 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-21T14:53:55.598 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-21T14:53:55.598 INFO:tasks.ceph:role: osd.0 2026-02-21T14:53:55.598 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:53:55.599 DEBUG:teuthology.orchestra.run.trial198:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-21T14:53:55.662 INFO:teuthology.orchestra.run.trial198.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout: = sunit=0 swidth=0 blks 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-21T14:53:55.663 INFO:teuthology.orchestra.run.trial198.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-21T14:53:56.231 INFO:teuthology.orchestra.run.trial198.stdout:Discarding blocks...Done. 2026-02-21T14:53:56.233 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial198.front.sepia.ceph.com -o noatime 2026-02-21T14:53:56.233 DEBUG:teuthology.orchestra.run.trial198:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-21T14:53:56.301 DEBUG:teuthology.orchestra.run.trial198:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-21T14:53:56.369 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-21T14:53:56.434 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-21T14:53:56.434 INFO:tasks.ceph:role: osd.1 2026-02-21T14:53:56.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:53:56.434 DEBUG:teuthology.orchestra.run.trial198:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-21T14:53:56.498 INFO:teuthology.orchestra.run.trial198.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout: = sunit=0 swidth=0 blks 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-21T14:53:56.499 INFO:teuthology.orchestra.run.trial198.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-21T14:53:57.057 INFO:teuthology.orchestra.run.trial198.stdout:Discarding blocks...Done. 2026-02-21T14:53:57.060 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial198.front.sepia.ceph.com -o noatime 2026-02-21T14:53:57.060 DEBUG:teuthology.orchestra.run.trial198:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-21T14:53:57.127 DEBUG:teuthology.orchestra.run.trial198:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-21T14:53:57.193 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-21T14:53:57.257 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-21T14:53:57.258 INFO:tasks.ceph:role: osd.2 2026-02-21T14:53:57.258 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:53:57.258 DEBUG:teuthology.orchestra.run.trial198:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-21T14:53:57.323 INFO:teuthology.orchestra.run.trial198.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-21T14:53:57.323 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-21T14:53:57.323 INFO:teuthology.orchestra.run.trial198.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-21T14:53:57.323 INFO:teuthology.orchestra.run.trial198.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-21T14:53:57.323 INFO:teuthology.orchestra.run.trial198.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-21T14:53:57.323 INFO:teuthology.orchestra.run.trial198.stdout: = sunit=0 swidth=0 blks 2026-02-21T14:53:57.324 INFO:teuthology.orchestra.run.trial198.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-21T14:53:57.324 INFO:teuthology.orchestra.run.trial198.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-21T14:53:57.324 INFO:teuthology.orchestra.run.trial198.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-21T14:53:57.324 INFO:teuthology.orchestra.run.trial198.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-21T14:53:57.880 INFO:teuthology.orchestra.run.trial198.stdout:Discarding blocks...Done. 2026-02-21T14:53:57.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial198.front.sepia.ceph.com -o noatime 2026-02-21T14:53:57.882 DEBUG:teuthology.orchestra.run.trial198:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-21T14:53:57.950 DEBUG:teuthology.orchestra.run.trial198:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-21T14:53:58.016 DEBUG:teuthology.orchestra.run.trial198:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-21T14:53:58.168 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: parsing early config 2026-02-21T14:53:58.220 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:53:58,231 seastar - Reactor backend: linux-aio 2026-02-21T14:53:58.231 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:53:58,242 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:53:58.231 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:53:58,242 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:53:58.234 INFO:teuthology.orchestra.run.trial198.stderr:==39357==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-21T14:53:58.290 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:53:58,300 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-21T14:53:58.290 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:53:58,300 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-21T14:53:58.290 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:53:58,300 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-21T14:53:58.290 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:53:58,300 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-21T14:53:58.290 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-21 14:53:58,301 [shard 0:main] seastar - reactor::drain 2026-02-21T14:54:00.396 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:00.396 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-21T14:54:00.396 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-21T14:54:00.396 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-21T14:54:00.397 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:00.397 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-21T14:54:00.415 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-21T14:54:00.415 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-21T14:54:00.460 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-21T14:54:00.460 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-21T14:54:00.467 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:00,478 seastar - Reactor backend: linux-aio 2026-02-21T14:54:00.477 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:00,488 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:00.477 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:00,488 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:00.479 INFO:teuthology.orchestra.run.trial198.stderr:==39355==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-21T14:54:00.480 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:00,490 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-21T14:54:00.538 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-21 14:54:00,550 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-21T14:54:00.616 INFO:teuthology.orchestra.run.trial198.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e4665c 0x9e46945 0x9ea72e3 0x9ea7ad6 0x9ea7d44 0x9ea7e8c 0x9ea834b /lib64/libc.so.6+0x3fc2f 0xf2ec67 0x9c6f140 0x9c6f43d 0x9c6f6b0 0x9c6f966 0x9c70415 0x9c70913 0x9c70f43 0x9c710ad 0x9d5141d 0x9d64220 0x9d26275 0x9d25275 0x7bb15cd 0x7bbb096 0x7a3b74b 0x7a7bab3 0x7111f37 0x6f927ce 0x6f965cf 0x6f9c179 0x74102d6 0x743293e 0x7282937 0x7282a2f 0x7353ce2 0x7353d9b 0x7353f52 0x74329bf 0x7282aac 0x7282b31 0x72ae77b 0x741b166 0x741b1eb 0x741b2c9 0x729a254 0x73a7d88 0x74274fc 0x7427577 0x74275f2 0x74eb1a2 0x75d925c 0x9f011ab 0x9e55480 0x9f2ec1b 0x9f973fe 0x9fb1e8e 0x9fb250a 0x26ec7eb 0xa5c0410 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-21T14:54:00.757 INFO:teuthology.orchestra.run.trial198.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e4665c 0x9e46945 0x9ea72e3 0x9ea7ad6 0x9ea7d44 0x9ea7e8c 0x9ea834b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbef /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x1126530 0xf2ec95 0x9c6ef73 0x9c6f43d 0x9c6f6b0 0x9c713d7 0x9c71924 0x9c71b23 0x9d56e45 0x9d575a0 0x9d6182e 0x9d65ca7 0x9d26275 0x9d25275 0x7ba0164 0x7bb5485 0x7bbb096 0x7a3b74b 0x7a7bab3 0x7111f37 0x6f3661e 0x6f38986 0x747af76 0x7299edd 0x7299fdf 0x729a0d7 0x7352eae 0x7352f67 0x735311a 0x7353252 0x73532cd 0x7353352 0x73534ad 0x741abe3 0x747afff 0x7281372 0x729a254 0x73a7d88 0x74274fc 0x7427577 0x74275f2 0x74eb1a2 0x75d925c 0x9f011ab 0x9e55480 0x9f2ec1b 0x9f973fe 0x9fb1e8e 0x9fb250a 0x26ec7eb 0xa5c0410 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-21T14:54:02.881 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:02.881 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-21T14:54:02.881 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-21T14:54:02.881 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-21T14:54:02.881 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:02.881 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-21T14:54:02.912 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-21T14:54:02.938 DEBUG:teuthology.orchestra.run.trial198:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-21T14:54:03.089 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: parsing early config 2026-02-21T14:54:03.149 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:03,160 seastar - Reactor backend: linux-aio 2026-02-21T14:54:03.161 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:03,172 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:03.161 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:03,172 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:03.163 INFO:teuthology.orchestra.run.trial198.stderr:==39419==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-21T14:54:03.219 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:03,230 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-21T14:54:03.219 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:03,230 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-21T14:54:03.219 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:03,230 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-21T14:54:03.219 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:03,230 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-21T14:54:03.219 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-21 14:54:03,230 [shard 0:main] seastar - reactor::drain 2026-02-21T14:54:05.298 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:05.298 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-21T14:54:05.298 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-21T14:54:05.298 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-21T14:54:05.298 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:05.298 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-21T14:54:05.317 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-21T14:54:05.318 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-21T14:54:05.367 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-21T14:54:05.367 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-21T14:54:05.374 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:05,385 seastar - Reactor backend: linux-aio 2026-02-21T14:54:05.384 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:05,395 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:05.384 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:05,395 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:05.387 INFO:teuthology.orchestra.run.trial198.stderr:==39417==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-21T14:54:05.387 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:05,398 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-21T14:54:05.444 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-21 14:54:05,455 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-21T14:54:05.522 INFO:teuthology.orchestra.run.trial198.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e4665c 0x9e46945 0x9ea72e3 0x9ea7ad6 0x9ea7d44 0x9ea7e8c 0x9ea834b /lib64/libc.so.6+0x3fc2f 0x7805e9f 0x7806254 0x9b436d6 0x9bc2bbd 0x9d39d08 0x9d39e01 0x7bb1541 0x7bbb096 0x7a3b74b 0x7a7bab3 0x7111f37 0x6f927ce 0x6f965cf 0x6f9c179 0x74102d6 0x743293e 0x7282937 0x7282a2f 0x7353ce2 0x7353d9b 0x7353f52 0x74329bf 0x7282aac 0x7282b31 0x72ae77b 0x741b166 0x741b1eb 0x741b2c9 0x729a254 0x73a7d88 0x74274fc 0x7427577 0x74275f2 0x74eb1a2 0x75d925c 0x9f011ab 0x9e55480 0x9f2ec1b 0x9f973fe 0x9fb1e8e 0x9fb250a 0x26ec7eb 0xa5c0410 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-21T14:54:05.664 INFO:teuthology.orchestra.run.trial198.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e4665c 0x9e46945 0x9ea72e3 0x9ea7ad6 0x9ea7d44 0x9ea7e8c 0x9ea834b /lib64/libc.so.6+0x3fc2f 0xb6f1cb 0x1091911 0xf2eca0 0x9c7399a 0x9c73c47 0x9d61139 0x9d65ca7 0x9d26275 0x9d25275 0x7ba0164 0x7bb52f1 0x7bbb096 0x7a3b74b 0x7a7bab3 0x7111f37 0x6f3661e 0x6f38986 0x747af76 0x7299edd 0x7299fdf 0x729a0d7 0x7352eae 0x7352f67 0x735311a 0x7353252 0x73532cd 0x7353352 0x73534ad 0x741abe3 0x747afff 0x7281372 0x729a254 0x73a7d88 0x74274fc 0x7427577 0x74275f2 0x74eb1a2 0x75d925c 0x9f011ab 0x9e55480 0x9f2ec1b 0x9f973fe 0x9fb1e8e 0x9fb250a 0x26ec7eb 0xa5c0410 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-21T14:54:07.779 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:07.779 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-21T14:54:07.779 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-21T14:54:07.779 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-21T14:54:07.780 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:07.780 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-21T14:54:07.810 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-21T14:54:07.835 DEBUG:teuthology.orchestra.run.trial198:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-21T14:54:07.987 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: parsing early config 2026-02-21T14:54:08.038 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:08,049 seastar - Reactor backend: linux-aio 2026-02-21T14:54:08.055 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:08,066 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:08.055 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:08,066 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:08.057 INFO:teuthology.orchestra.run.trial198.stderr:==39481==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-21T14:54:08.113 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:08,124 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-21T14:54:08.113 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:08,124 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-21T14:54:08.113 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:08,124 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-21T14:54:08.113 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:08,124 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-21T14:54:08.114 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-21 14:54:08,124 [shard 0:main] seastar - reactor::drain 2026-02-21T14:54:10.211 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:10.211 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-21T14:54:10.211 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-21T14:54:10.211 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-21T14:54:10.211 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:10.211 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-21T14:54:10.230 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-21T14:54:10.231 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-21T14:54:10.272 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-21T14:54:10.272 INFO:teuthology.orchestra.run.trial198.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-21T14:54:10.279 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:10,290 seastar - Reactor backend: linux-aio 2026-02-21T14:54:10.293 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:10,304 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:10.293 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:10,304 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-21T14:54:10.296 INFO:teuthology.orchestra.run.trial198.stderr:==39479==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-21T14:54:10.296 INFO:teuthology.orchestra.run.trial198.stderr:INFO 2026-02-21 14:54:10,307 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-21T14:54:10.353 INFO:teuthology.orchestra.run.trial198.stderr:DEBUG 2026-02-21 14:54:10,364 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-21T14:54:10.431 INFO:teuthology.orchestra.run.trial198.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e4665c 0x9e46945 0x9ea72e3 0x9ea7ad6 0x9ea7d44 0x9ea7e8c 0x9ea834b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x3423 /lib64/libasan.so.8+0x15197 0x1126530 0xf2ec95 0x9c6ef73 0x9c6f43d 0x9c6f6b0 0x9c713d7 0x9c71924 0x9c71b23 0x9d56e45 0x9d575a0 0x9d6182e 0x9d65ca7 0x9d26275 0x9d25275 0x7bb15cd 0x7bbb096 0x7a3b74b 0x7a7bab3 0x7111f37 0x6f927ce 0x6f965cf 0x6f9c179 0x74102d6 0x743293e 0x7282937 0x7282a2f 0x7353ce2 0x7353d9b 0x7353f52 0x74329bf 0x7282aac 0x7282b31 0x72ae77b 0x741b166 0x741b1eb 0x741b2c9 0x729a254 0x73a7d88 0x74274fc 0x7427577 0x74275f2 0x74eb1a2 0x75d925c 0x9f011ab 0x9e55480 0x9f2ec1b 0x9f973fe 0x9fb1e8e 0x9fb250a 0x26ec7eb 0xa5c0410 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-21T14:54:10.572 INFO:teuthology.orchestra.run.trial198.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e4665c 0x9e46945 0x9ea72e3 0x9ea7ad6 0x9ea7d44 0x9ea7e8c 0x9ea834b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xeaac2 0xe1de8e 0xe1e249 0x9d6449c 0x9d26275 0x9d25275 0x7ba0164 0x7bb52f1 0x7bbb096 0x7a3b74b 0x7a7bab3 0x7111f37 0x6f3661e 0x6f38986 0x747af76 0x7299edd 0x7299fdf 0x729a0d7 0x7352eae 0x7352f67 0x735311a 0x7353252 0x73532cd 0x7353352 0x73534ad 0x741abe3 0x747afff 0x7281372 0x729a254 0x73a7d88 0x74274fc 0x7427577 0x74275f2 0x74eb1a2 0x75d925c 0x9f011ab 0x9e55480 0x9f2ec1b 0x9f973fe 0x9fb1e8e 0x9fb250a 0x26ec7eb 0xa5c0410 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-21T14:54:12.743 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:12.743 INFO:teuthology.orchestra.run.trial198.stderr:Suppressions used: 2026-02-21T14:54:12.743 INFO:teuthology.orchestra.run.trial198.stderr: count bytes template 2026-02-21T14:54:12.743 INFO:teuthology.orchestra.run.trial198.stderr: 1 8 MallocExtension::Register 2026-02-21T14:54:12.744 INFO:teuthology.orchestra.run.trial198.stderr:----------------------------------------------------- 2026-02-21T14:54:12.744 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-21T14:54:12.773 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-21T14:54:12.798 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-21T14:54:12.798 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:12.798 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-21T14:54:12.862 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:12.862 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-21T14:54:12.925 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:12.925 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-21T14:54:12.990 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:12.990 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-21T14:54:13.054 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:13.054 DEBUG:teuthology.orchestra.run.trial198:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-21T14:54:13.110 INFO:tasks.ceph:Adding keys to all mons... 2026-02-21T14:54:13.110 DEBUG:teuthology.orchestra.run.trial198:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-21T14:54:13.173 INFO:teuthology.orchestra.run.trial198.stdout:[mgr.x] 2026-02-21T14:54:13.173 INFO:teuthology.orchestra.run.trial198.stdout: key = AQADx5lp7fkJHRAABPXJIVFZu3ksrxMNg0LWww== 2026-02-21T14:54:13.173 INFO:teuthology.orchestra.run.trial198.stdout:[osd.0] 2026-02-21T14:54:13.173 INFO:teuthology.orchestra.run.trial198.stdout: key = AQAIx5lpKWEDIRAAozEMKq6AWv8Qs5wKl+WNSw== 2026-02-21T14:54:13.173 INFO:teuthology.orchestra.run.trial198.stdout:[osd.1] 2026-02-21T14:54:13.173 INFO:teuthology.orchestra.run.trial198.stdout: key = AQANx5lpnKxiGxAAvc8WK0JC20Zy5DCit8d4pw== 2026-02-21T14:54:13.174 INFO:teuthology.orchestra.run.trial198.stdout:[osd.2] 2026-02-21T14:54:13.174 INFO:teuthology.orchestra.run.trial198.stdout: key = AQASx5lptx/4FRAALHExJ0WyNEzfFuR169rInA== 2026-02-21T14:54:13.174 INFO:teuthology.orchestra.run.trial198.stdout:[client.0] 2026-02-21T14:54:13.174 INFO:teuthology.orchestra.run.trial198.stdout: key = AQADx5lpLJK6HxAAJ6DzUJcMN267RWWpMa/Qkg== 2026-02-21T14:54:13.175 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-21T14:54:13.262 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-21T14:54:13.310 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-21T14:54:13.358 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-21T14:54:13.407 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-21T14:54:13.459 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-21T14:54:13.459 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-21T14:54:13.484 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-21T14:54:13.570 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:13.581+0000 7f5d9eed8c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:13.571 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:13.581+0000 7f5d9eed8c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:13.582 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-21T14:54:13.606 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-21T14:54:13.671 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-21T14:54:13.755 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:13.766+0000 7fa9e9ab2c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:13.756 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:13.766+0000 7fa9e9ab2c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:13.767 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-21T14:54:13.792 DEBUG:teuthology.orchestra.run.trial198:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-21T14:54:13.856 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-21T14:54:13.939 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:13.950+0000 7f66896a5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:13.940 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:13.950+0000 7f66896a5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:13.951 DEBUG:teuthology.orchestra.run.trial198:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-21T14:54:13.977 DEBUG:teuthology.orchestra.run.trial198:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-21T14:54:14.033 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-21T14:54:14.033 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-21T14:54:14.033 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-21T14:54:14.075 INFO:tasks.ceph.mon.a:Started 2026-02-21T14:54:14.075 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-21T14:54:14.075 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-21T14:54:14.077 INFO:tasks.ceph.mon.b:Started 2026-02-21T14:54:14.077 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-21T14:54:14.078 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-21T14:54:14.080 INFO:tasks.ceph.mon.c:Started 2026-02-21T14:54:14.080 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph config set mgr mgr_pool false --force 2026-02-21T14:54:14.136 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-21T14:54:14.146+0000 7fc7b7889c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.136 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-21T14:54:14.146+0000 7fc7b7889c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.136 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-21T14:54:14.146+0000 7fc7b7889c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.140 INFO:tasks.ceph.mon.b.trial198.stderr:2026-02-21T14:54:14.150+0000 7f0c53081c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.140 INFO:tasks.ceph.mon.b.trial198.stderr:2026-02-21T14:54:14.150+0000 7f0c53081c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.140 INFO:tasks.ceph.mon.b.trial198.stderr:2026-02-21T14:54:14.150+0000 7f0c53081c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.143 INFO:tasks.ceph.mon.c.trial198.stderr:2026-02-21T14:54:14.153+0000 7ff245c94c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.143 INFO:tasks.ceph.mon.c.trial198.stderr:2026-02-21T14:54:14.153+0000 7ff245c94c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.143 INFO:tasks.ceph.mon.c.trial198.stderr:2026-02-21T14:54:14.153+0000 7ff245c94c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.158 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.163+0000 7faa61411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.365 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.375+0000 7faa61411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.411 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-21T14:54:14.411 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-21T14:54:14.411 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-21T14:54:14.413 INFO:tasks.ceph.mgr.x:Started 2026-02-21T14:54:14.414 DEBUG:tasks.ceph:set 0 configs 2026-02-21T14:54:14.414 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph config dump 2026-02-21T14:54:14.475 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-21T14:54:14.485+0000 7f76f79c3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.478 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-21T14:54:14.488+0000 7f76f79c3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.478 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-21T14:54:14.488+0000 7f76f79c3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.487 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.496+0000 7f8eff59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.490 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.501+0000 7f8eff59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.523 INFO:teuthology.orchestra.run.trial198.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-21T14:54:14.523 INFO:teuthology.orchestra.run.trial198.stdout:mgr dev mgr_pool false * 2026-02-21T14:54:14.532 INFO:tasks.ceph:Setting crush tunables to default 2026-02-21T14:54:14.532 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd crush tunables default 2026-02-21T14:54:14.607 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.613+0000 7f8bf02cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.610 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.620+0000 7f8bf02cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.647 INFO:teuthology.orchestra.run.trial198.stderr:adjusted tunables profile to default 2026-02-21T14:54:14.655 INFO:tasks.ceph:check_enable_crimson: True 2026-02-21T14:54:14.656 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-21T14:54:14.656 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-21T14:54:14.728 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.735+0000 7fe6ccd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.730 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:14.741+0000 7fe6ccd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:14.770 INFO:tasks.ceph.mon.a.trial198.stderr:2026-02-21T14:54:14.780+0000 7fc7aeff1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-21T14:54:14.770 INFO:tasks.ceph.mon.a.trial198.stderr:Please be aware that this feature is experimental, untested, 2026-02-21T14:54:14.770 INFO:tasks.ceph.mon.a.trial198.stderr:unsupported, and may result in data corruption, data loss, 2026-02-21T14:54:14.770 INFO:tasks.ceph.mon.a.trial198.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-21T14:54:14.770 INFO:tasks.ceph.mon.a.trial198.stderr:feature with important data. 2026-02-21T14:54:14.770 INFO:tasks.ceph.mon.a.trial198.stderr: 2026-02-21T14:54:15.312 INFO:tasks.ceph.mgr.x.trial198.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-21T14:54:15.312 INFO:tasks.ceph.mgr.x.trial198.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-21T14:54:15.313 INFO:tasks.ceph.mgr.x.trial198.stderr: from numpy import show_config as show_numpy_config 2026-02-21T14:54:15.658 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-21T14:54:15.659 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:15.659 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-21T14:54:15.684 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:15.684 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-21T14:54:15.747 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:15.747 DEBUG:teuthology.orchestra.run.trial198:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-21T14:54:15.811 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd new dd060cd1-0fe5-49de-b464-5d41b744ed56 0 2026-02-21T14:54:15.926 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:15.933+0000 7fa105ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:15.928 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:15.939+0000 7fa105ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:15.968 INFO:teuthology.orchestra.run.trial198.stdout:0 2026-02-21T14:54:15.976 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd new 7573fcca-b139-4885-83fe-e9233d285f58 1 2026-02-21T14:54:16.050 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:16.057+0000 7f92f870b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.052 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:16.063+0000 7f92f870b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.092 INFO:teuthology.orchestra.run.trial198.stdout:1 2026-02-21T14:54:16.101 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd new a0db30db-8a0b-4baa-ae29-0976e2909a6a 2 2026-02-21T14:54:16.174 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:16.181+0000 7f8fb5ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.177 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:16.188+0000 7f8fb5ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.218 INFO:teuthology.orchestra.run.trial198.stdout:2 2026-02-21T14:54:16.226 DEBUG:teuthology.orchestra.run.trial198:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-21T14:54:16.253 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-21T14:54:16.254 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-21T14:54:16.296 INFO:tasks.ceph.osd.0:Started 2026-02-21T14:54:16.296 DEBUG:teuthology.orchestra.run.trial198:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-21T14:54:16.324 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-21T14:54:16.324 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-21T14:54:16.365 INFO:tasks.ceph.osd.1:Started 2026-02-21T14:54:16.366 DEBUG:teuthology.orchestra.run.trial198:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-21T14:54:16.392 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-21T14:54:16.393 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-21T14:54:16.436 INFO:tasks.ceph.osd.2:Started 2026-02-21T14:54:16.436 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-21T14:54:16.509 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:16.518+0000 7f3a8e30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.512 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:16.523+0000 7f3a8e30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.546 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:16.547 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":6,"fsid":"7ba0c362-e064-4d8f-9e0d-89e7be54cc27","created":"2026-02-21T14:54:14.367697+0000","modified":"2026-02-21T14:54:16.226959+0000","last_up_change":"0.000000","last_in_change":"2026-02-21T14:54:16.226959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd060cd1-0fe5-49de-b464-5d41b744ed56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7573fcca-b139-4885-83fe-e9233d285f58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a0db30db-8a0b-4baa-ae29-0976e2909a6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-21T14:54:16.555 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-21T14:54:16.555 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-21T14:54:16.856 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-21T14:54:16.923 INFO:teuthology.misc.health.trial198.stderr:2026-02-21T14:54:16.931+0000 7f6999f74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.926 INFO:teuthology.misc.health.trial198.stderr:2026-02-21T14:54:16.936+0000 7f6999f74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:16.959 INFO:teuthology.misc.health.trial198.stdout: 2026-02-21T14:54:16.959 INFO:teuthology.misc.health.trial198.stdout:{"epoch":6,"fsid":"7ba0c362-e064-4d8f-9e0d-89e7be54cc27","created":"2026-02-21T14:54:14.367697+0000","modified":"2026-02-21T14:54:16.226959+0000","last_up_change":"0.000000","last_in_change":"2026-02-21T14:54:16.226959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd060cd1-0fe5-49de-b464-5d41b744ed56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7573fcca-b139-4885-83fe-e9233d285f58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a0db30db-8a0b-4baa-ae29-0976e2909a6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-21T14:54:16.966 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-21T14:54:17.699 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-21T14:54:17.710+0000 7f7612873640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-21T14:54:17.700 INFO:tasks.ceph.mgr.x.trial198.stderr:2026-02-21T14:54:17.710+0000 7f7612873640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-21T14:54:23.268 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-21T14:54:23.336 INFO:teuthology.misc.health.trial198.stderr:2026-02-21T14:54:23.345+0000 7f7ae685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:23.339 INFO:teuthology.misc.health.trial198.stderr:2026-02-21T14:54:23.349+0000 7f7ae685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:23.454 INFO:teuthology.misc.health.trial198.stdout: 2026-02-21T14:54:23.454 INFO:teuthology.misc.health.trial198.stdout:{"epoch":7,"fsid":"7ba0c362-e064-4d8f-9e0d-89e7be54cc27","created":"2026-02-21T14:54:14.367697+0000","modified":"2026-02-21T14:54:22.717768+0000","last_up_change":"0.000000","last_in_change":"2026-02-21T14:54:16.226959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd060cd1-0fe5-49de-b464-5d41b744ed56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7573fcca-b139-4885-83fe-e9233d285f58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a0db30db-8a0b-4baa-ae29-0976e2909a6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-21T14:54:23.462 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-21T14:54:29.764 DEBUG:teuthology.orchestra.run.trial198:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-21T14:54:29.831 INFO:teuthology.misc.health.trial198.stderr:2026-02-21T14:54:29.838+0000 7ff17edb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:29.834 INFO:teuthology.misc.health.trial198.stderr:2026-02-21T14:54:29.843+0000 7ff17edb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:29.947 INFO:teuthology.misc.health.trial198.stdout: 2026-02-21T14:54:29.947 INFO:teuthology.misc.health.trial198.stdout:{"epoch":10,"fsid":"7ba0c362-e064-4d8f-9e0d-89e7be54cc27","created":"2026-02-21T14:54:14.367697+0000","modified":"2026-02-21T14:54:25.728036+0000","last_up_change":"2026-02-21T14:54:25.728036+0000","last_in_change":"2026-02-21T14:54:16.226959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd060cd1-0fe5-49de-b464-5d41b744ed56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":3318032325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3318032325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3318032325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6814","nonce":3318032325}]},"public_addr":"10.20.193.198:6804/3318032325","cluster_addr":"10.20.193.198:6805/3318032325","heartbeat_back_addr":"10.20.193.198:6811/3318032325","heartbeat_front_addr":"10.20.193.198:6814/3318032325","state":["exists","up"]},{"osd":1,"uuid":"7573fcca-b139-4885-83fe-e9233d285f58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1811593807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1811593807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":1811593807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":1811593807}]},"public_addr":"10.20.193.198:6803/1811593807","cluster_addr":"10.20.193.198:6802/1811593807","heartbeat_back_addr":"10.20.193.198:6809/1811593807","heartbeat_front_addr":"10.20.193.198:6806/1811593807","state":["exists","up"]},{"osd":2,"uuid":"a0db30db-8a0b-4baa-ae29-0976e2909a6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":746110398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":746110398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":746110398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":746110398}]},"public_addr":"10.20.193.198:6807/746110398","cluster_addr":"10.20.193.198:6808/746110398","heartbeat_back_addr":"10.20.193.198:6813/746110398","heartbeat_front_addr":"10.20.193.198:6812/746110398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-21T14:54:29.955 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-21T14:54:29.955 INFO:tasks.ceph:Creating RBD pool 2026-02-21T14:54:29.956 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-21T14:54:30.029 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:30.035+0000 7f4f9e4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:30.031 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:30.041+0000 7f4f9e4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:31.003 INFO:teuthology.orchestra.run.trial198.stderr:pool 'rbd' created 2026-02-21T14:54:31.017 DEBUG:teuthology.orchestra.run.trial198:> rbd --cluster ceph pool init rbd 2026-02-21T14:54:31.059 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:31.068+0000 7f3c45e0e240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:31.059 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:31.068+0000 7f3c45e0e240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:31.062 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:31.072+0000 7f3c45e0e240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.048 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-21T14:54:34.049 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-21T14:54:34.049 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-21T14:54:34.163 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.169+0000 7f0a0bbad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.166 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.175+0000 7f0a0bbad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.279 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:34.288 INFO:teuthology.orchestra.run.trial198.stdout:[{"version":2,"timestamp":"2026-02-21T14:54:14.411131+0000","name":"","changes":[{"name":"mgr/mgr_pool","new_value":"false"}]}] 2026-02-21T14:54:34.288 INFO:tasks.ceph_manager:config epoch is 2 2026-02-21T14:54:34.288 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-21T14:54:34.288 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-21T14:54:34.289 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-21T14:54:34.363 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.371+0000 7f2a699b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.366 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.375+0000 7f2a699b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.513 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:34.521 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6800","nonce":1367640693},{"type":"v1","addr":"10.20.193.198:6801","nonce":1367640693}]},"active_addr":"10.20.193.198:6801/1367640693","active_change":"2026-02-21T14:54:16.702143+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.198:0","nonce":91030440}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.198:0","nonce":2572972713}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.198:0","nonce":3003194335}]}]} 2026-02-21T14:54:34.525 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-21T14:54:34.525 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-21T14:54:34.525 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-21T14:54:34.601 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.608+0000 7faa9383d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.603 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.613+0000 7faa9383d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.717 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:34.717 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":14,"fsid":"7ba0c362-e064-4d8f-9e0d-89e7be54cc27","created":"2026-02-21T14:54:14.367697+0000","modified":"2026-02-21T14:54:34.022284+0000","last_up_change":"2026-02-21T14:54:25.728036+0000","last_in_change":"2026-02-21T14:54:16.226959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-21T14:54:30.163921+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd060cd1-0fe5-49de-b464-5d41b744ed56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":3318032325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3318032325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3318032325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6814","nonce":3318032325}]},"public_addr":"10.20.193.198:6804/3318032325","cluster_addr":"10.20.193.198:6805/3318032325","heartbeat_back_addr":"10.20.193.198:6811/3318032325","heartbeat_front_addr":"10.20.193.198:6814/3318032325","state":["exists","up"]},{"osd":1,"uuid":"7573fcca-b139-4885-83fe-e9233d285f58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1811593807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1811593807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":1811593807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":1811593807}]},"public_addr":"10.20.193.198:6803/1811593807","cluster_addr":"10.20.193.198:6802/1811593807","heartbeat_back_addr":"10.20.193.198:6809/1811593807","heartbeat_front_addr":"10.20.193.198:6806/1811593807","state":["exists","up"]},{"osd":2,"uuid":"a0db30db-8a0b-4baa-ae29-0976e2909a6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":746110398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":746110398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":746110398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":746110398}]},"public_addr":"10.20.193.198:6807/746110398","cluster_addr":"10.20.193.198:6808/746110398","heartbeat_back_addr":"10.20.193.198:6813/746110398","heartbeat_front_addr":"10.20.193.198:6812/746110398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-21T14:54:34.727 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-21T14:54:34.727 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-21T14:54:34.802 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.809+0000 7fcdfff0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.804 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:34.814+0000 7fcdfff0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:34.918 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:34.918 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":14,"fsid":"7ba0c362-e064-4d8f-9e0d-89e7be54cc27","created":"2026-02-21T14:54:14.367697+0000","modified":"2026-02-21T14:54:34.022284+0000","last_up_change":"2026-02-21T14:54:25.728036+0000","last_in_change":"2026-02-21T14:54:16.226959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-21T14:54:30.163921+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd060cd1-0fe5-49de-b464-5d41b744ed56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":3318032325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3318032325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3318032325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6814","nonce":3318032325}]},"public_addr":"10.20.193.198:6804/3318032325","cluster_addr":"10.20.193.198:6805/3318032325","heartbeat_back_addr":"10.20.193.198:6811/3318032325","heartbeat_front_addr":"10.20.193.198:6814/3318032325","state":["exists","up"]},{"osd":1,"uuid":"7573fcca-b139-4885-83fe-e9233d285f58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1811593807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1811593807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":1811593807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":1811593807}]},"public_addr":"10.20.193.198:6803/1811593807","cluster_addr":"10.20.193.198:6802/1811593807","heartbeat_back_addr":"10.20.193.198:6809/1811593807","heartbeat_front_addr":"10.20.193.198:6806/1811593807","state":["exists","up"]},{"osd":2,"uuid":"a0db30db-8a0b-4baa-ae29-0976e2909a6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":746110398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":746110398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":746110398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":746110398}]},"public_addr":"10.20.193.198:6807/746110398","cluster_addr":"10.20.193.198:6808/746110398","heartbeat_back_addr":"10.20.193.198:6813/746110398","heartbeat_front_addr":"10.20.193.198:6812/746110398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-21T14:54:34.928 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-21T14:54:34.929 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-21T14:54:34.929 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-21T14:54:35.009 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.015+0000 7f3842acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.009 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.015+0000 7f01800cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.010 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.015+0000 7f7a2fab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.012 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.021+0000 7f3842acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.012 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.021+0000 7f7a2fab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.012 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.021+0000 7f01800cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.032 INFO:teuthology.orchestra.run.trial198.stdout:42949672969 2026-02-21T14:54:35.033 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-21T14:54:35.033 INFO:teuthology.orchestra.run.trial198.stdout:42949672969 2026-02-21T14:54:35.034 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-21T14:54:35.035 INFO:teuthology.orchestra.run.trial198.stdout:38654705674 2026-02-21T14:54:35.035 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-21T14:54:35.153 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.159+0000 7f212b1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.154 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.160+0000 7fb31545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.154 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.160+0000 7fd72719c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.156 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.165+0000 7fb31545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.156 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.166+0000 7f212b1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.157 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:35.166+0000 7fd72719c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:35.273 INFO:teuthology.orchestra.run.trial198.stdout:42949672964 2026-02-21T14:54:35.273 INFO:teuthology.orchestra.run.trial198.stdout:42949672964 2026-02-21T14:54:35.276 INFO:teuthology.orchestra.run.trial198.stdout:38654705668 2026-02-21T14:54:35.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.2 2026-02-21T14:54:35.285 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.0 2026-02-21T14:54:35.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705668 for osd.1 2026-02-21T14:54:36.285 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-21T14:54:36.285 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-21T14:54:36.289 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-21T14:54:36.363 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:36.369+0000 7f6e1508b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:36.364 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:36.370+0000 7faa406cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:36.366 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:36.375+0000 7f6e1508b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:36.366 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:36.376+0000 7faa406cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:36.403 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:36.409+0000 7eff8405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:36.406 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:36.415+0000 7eff8405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:36.482 INFO:teuthology.orchestra.run.trial198.stdout:42949672964 2026-02-21T14:54:36.483 INFO:teuthology.orchestra.run.trial198.stdout:42949672964 2026-02-21T14:54:36.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.0 2026-02-21T14:54:36.495 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.2 2026-02-21T14:54:36.521 INFO:teuthology.orchestra.run.trial198.stdout:38654705668 2026-02-21T14:54:36.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705668 for osd.1 2026-02-21T14:54:37.494 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-21T14:54:37.495 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-21T14:54:37.532 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-21T14:54:37.575 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.581+0000 7f129145a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.576 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.582+0000 7f75263a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.578 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.588+0000 7f75263a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.579 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.588+0000 7f129145a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.614 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.620+0000 7f8e1ae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.617 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.627+0000 7f8e1ae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.694 INFO:teuthology.orchestra.run.trial198.stdout:42949672969 2026-02-21T14:54:37.696 INFO:teuthology.orchestra.run.trial198.stdout:42949672969 2026-02-21T14:54:37.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-21T14:54:37.705 DEBUG:teuthology.parallel:result is None 2026-02-21T14:54:37.708 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-21T14:54:37.709 DEBUG:teuthology.parallel:result is None 2026-02-21T14:54:37.734 INFO:teuthology.orchestra.run.trial198.stdout:38654705674 2026-02-21T14:54:37.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-21T14:54:37.744 DEBUG:teuthology.parallel:result is None 2026-02-21T14:54:37.745 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-21T14:54:37.745 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-21T14:54:37.859 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.867+0000 7f11bbd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.862 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:37.871+0000 7f11bbd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:37.977 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:37.977 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-21T14:54:37.986 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-21T14:54:36.707280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286376,"kb_used":31458060,"kb_used_data":31458060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125828316,"statfs":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.040844+0000","last_change":"2026-02-21T14:54:34.050037+0000","last_active":"2026-02-21T14:54:34.040844+0000","last_peered":"2026-02-21T14:54:34.040844+0000","last_clean":"2026-02-21T14:54:34.040844+0000","last_became_active":"2026-02-21T14:54:32.070503+0000","last_became_peered":"2026-02-21T14:54:32.070503+0000","last_unstale":"2026-02-21T14:54:34.040844+0000","last_undegraded":"2026-02-21T14:54:34.040844+0000","last_fullsized":"2026-02-21T14:54:34.040844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037776+0000","last_change":"2026-02-21T14:54:34.044270+0000","last_active":"2026-02-21T14:54:34.037776+0000","last_peered":"2026-02-21T14:54:34.037776+0000","last_clean":"2026-02-21T14:54:34.037776+0000","last_became_active":"2026-02-21T14:54:32.079015+0000","last_became_peered":"2026-02-21T14:54:32.079015+0000","last_unstale":"2026-02-21T14:54:34.037776+0000","last_undegraded":"2026-02-21T14:54:34.037776+0000","last_fullsized":"2026-02-21T14:54:34.037776+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038816+0000","last_change":"2026-02-21T14:54:34.042321+0000","last_active":"2026-02-21T14:54:34.038816+0000","last_peered":"2026-02-21T14:54:34.038816+0000","last_clean":"2026-02-21T14:54:34.038816+0000","last_became_active":"2026-02-21T14:54:32.133652+0000","last_became_peered":"2026-02-21T14:54:32.133652+0000","last_unstale":"2026-02-21T14:54:34.038816+0000","last_undegraded":"2026-02-21T14:54:34.038816+0000","last_fullsized":"2026-02-21T14:54:34.038816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041973+0000","last_change":"2026-02-21T14:54:34.050506+0000","last_active":"2026-02-21T14:54:34.041973+0000","last_peered":"2026-02-21T14:54:34.041973+0000","last_clean":"2026-02-21T14:54:34.041973+0000","last_became_active":"2026-02-21T14:54:32.111585+0000","last_became_peered":"2026-02-21T14:54:32.111585+0000","last_unstale":"2026-02-21T14:54:34.041973+0000","last_undegraded":"2026-02-21T14:54:34.041973+0000","last_fullsized":"2026-02-21T14:54:34.041973+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.042316+0000","last_change":"2026-02-21T14:54:34.050474+0000","last_active":"2026-02-21T14:54:34.042316+0000","last_peered":"2026-02-21T14:54:34.042316+0000","last_clean":"2026-02-21T14:54:34.042316+0000","last_became_active":"2026-02-21T14:54:32.074657+0000","last_became_peered":"2026-02-21T14:54:32.074657+0000","last_unstale":"2026-02-21T14:54:34.042316+0000","last_undegraded":"2026-02-21T14:54:34.042316+0000","last_fullsized":"2026-02-21T14:54:34.042316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041371+0000","last_change":"2026-02-21T14:54:34.041371+0000","last_active":"2026-02-21T14:54:34.041371+0000","last_peered":"2026-02-21T14:54:34.041371+0000","last_clean":"2026-02-21T14:54:34.041371+0000","last_became_active":"2026-02-21T14:54:32.138176+0000","last_became_peered":"2026-02-21T14:54:32.138176+0000","last_unstale":"2026-02-21T14:54:34.041371+0000","last_undegraded":"2026-02-21T14:54:34.041371+0000","last_fullsized":"2026-02-21T14:54:34.041371+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037447+0000","last_change":"2026-02-21T14:54:34.041349+0000","last_active":"2026-02-21T14:54:34.037447+0000","last_peered":"2026-02-21T14:54:34.037447+0000","last_clean":"2026-02-21T14:54:34.037447+0000","last_became_active":"2026-02-21T14:54:32.100414+0000","last_became_peered":"2026-02-21T14:54:32.100414+0000","last_unstale":"2026-02-21T14:54:34.037447+0000","last_undegraded":"2026-02-21T14:54:34.037447+0000","last_fullsized":"2026-02-21T14:54:34.037447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038019+0000","last_change":"2026-02-21T14:54:34.044647+0000","last_active":"2026-02-21T14:54:34.038019+0000","last_peered":"2026-02-21T14:54:34.038019+0000","last_clean":"2026-02-21T14:54:34.038019+0000","last_became_active":"2026-02-21T14:54:32.091454+0000","last_became_peered":"2026-02-21T14:54:32.091454+0000","last_unstale":"2026-02-21T14:54:34.038019+0000","last_undegraded":"2026-02-21T14:54:34.038019+0000","last_fullsized":"2026-02-21T14:54:34.038019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10485996,"kb_used_data":10485996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942796,"statfs":{"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-21T14:54:37.987 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-21T14:54:38.063 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:38.070+0000 7f7bdbdb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:38.066 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:38.075+0000 7f7bdbdb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:38.179 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:38.180 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-21T14:54:38.189 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-21T14:54:36.707280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286376,"kb_used":31458060,"kb_used_data":31458060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125828316,"statfs":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.040844+0000","last_change":"2026-02-21T14:54:34.050037+0000","last_active":"2026-02-21T14:54:34.040844+0000","last_peered":"2026-02-21T14:54:34.040844+0000","last_clean":"2026-02-21T14:54:34.040844+0000","last_became_active":"2026-02-21T14:54:32.070503+0000","last_became_peered":"2026-02-21T14:54:32.070503+0000","last_unstale":"2026-02-21T14:54:34.040844+0000","last_undegraded":"2026-02-21T14:54:34.040844+0000","last_fullsized":"2026-02-21T14:54:34.040844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037776+0000","last_change":"2026-02-21T14:54:34.044270+0000","last_active":"2026-02-21T14:54:34.037776+0000","last_peered":"2026-02-21T14:54:34.037776+0000","last_clean":"2026-02-21T14:54:34.037776+0000","last_became_active":"2026-02-21T14:54:32.079015+0000","last_became_peered":"2026-02-21T14:54:32.079015+0000","last_unstale":"2026-02-21T14:54:34.037776+0000","last_undegraded":"2026-02-21T14:54:34.037776+0000","last_fullsized":"2026-02-21T14:54:34.037776+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038816+0000","last_change":"2026-02-21T14:54:34.042321+0000","last_active":"2026-02-21T14:54:34.038816+0000","last_peered":"2026-02-21T14:54:34.038816+0000","last_clean":"2026-02-21T14:54:34.038816+0000","last_became_active":"2026-02-21T14:54:32.133652+0000","last_became_peered":"2026-02-21T14:54:32.133652+0000","last_unstale":"2026-02-21T14:54:34.038816+0000","last_undegraded":"2026-02-21T14:54:34.038816+0000","last_fullsized":"2026-02-21T14:54:34.038816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041973+0000","last_change":"2026-02-21T14:54:34.050506+0000","last_active":"2026-02-21T14:54:34.041973+0000","last_peered":"2026-02-21T14:54:34.041973+0000","last_clean":"2026-02-21T14:54:34.041973+0000","last_became_active":"2026-02-21T14:54:32.111585+0000","last_became_peered":"2026-02-21T14:54:32.111585+0000","last_unstale":"2026-02-21T14:54:34.041973+0000","last_undegraded":"2026-02-21T14:54:34.041973+0000","last_fullsized":"2026-02-21T14:54:34.041973+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.042316+0000","last_change":"2026-02-21T14:54:34.050474+0000","last_active":"2026-02-21T14:54:34.042316+0000","last_peered":"2026-02-21T14:54:34.042316+0000","last_clean":"2026-02-21T14:54:34.042316+0000","last_became_active":"2026-02-21T14:54:32.074657+0000","last_became_peered":"2026-02-21T14:54:32.074657+0000","last_unstale":"2026-02-21T14:54:34.042316+0000","last_undegraded":"2026-02-21T14:54:34.042316+0000","last_fullsized":"2026-02-21T14:54:34.042316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041371+0000","last_change":"2026-02-21T14:54:34.041371+0000","last_active":"2026-02-21T14:54:34.041371+0000","last_peered":"2026-02-21T14:54:34.041371+0000","last_clean":"2026-02-21T14:54:34.041371+0000","last_became_active":"2026-02-21T14:54:32.138176+0000","last_became_peered":"2026-02-21T14:54:32.138176+0000","last_unstale":"2026-02-21T14:54:34.041371+0000","last_undegraded":"2026-02-21T14:54:34.041371+0000","last_fullsized":"2026-02-21T14:54:34.041371+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037447+0000","last_change":"2026-02-21T14:54:34.041349+0000","last_active":"2026-02-21T14:54:34.037447+0000","last_peered":"2026-02-21T14:54:34.037447+0000","last_clean":"2026-02-21T14:54:34.037447+0000","last_became_active":"2026-02-21T14:54:32.100414+0000","last_became_peered":"2026-02-21T14:54:32.100414+0000","last_unstale":"2026-02-21T14:54:34.037447+0000","last_undegraded":"2026-02-21T14:54:34.037447+0000","last_fullsized":"2026-02-21T14:54:34.037447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038019+0000","last_change":"2026-02-21T14:54:34.044647+0000","last_active":"2026-02-21T14:54:34.038019+0000","last_peered":"2026-02-21T14:54:34.038019+0000","last_clean":"2026-02-21T14:54:34.038019+0000","last_became_active":"2026-02-21T14:54:32.091454+0000","last_became_peered":"2026-02-21T14:54:32.091454+0000","last_unstale":"2026-02-21T14:54:34.038019+0000","last_undegraded":"2026-02-21T14:54:34.038019+0000","last_fullsized":"2026-02-21T14:54:34.038019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10485996,"kb_used_data":10485996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942796,"statfs":{"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-21T14:54:38.190 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-21T14:54:38.190 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-21T14:54:38.190 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-21T14:54:38.190 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-21T14:54:38.266 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:38.273+0000 7f6a1357e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:38.268 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:38.278+0000 7f6a1357e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:38.408 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:38.408 INFO:teuthology.orchestra.run.trial198.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-21T14:54:38.417 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-21T14:54:38.418 INFO:teuthology.run_tasks:Running task workunit... 2026-02-21T14:54:38.443 INFO:tasks.workunit:Pulling workunits from ref c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f 2026-02-21T14:54:38.443 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-21T14:54:38.444 DEBUG:teuthology.orchestra.run.trial198:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-21T14:54:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:54:38.461 INFO:teuthology.orchestra.run.trial198.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-21T14:54:38.461 DEBUG:teuthology.orchestra.run.trial198:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-21T14:54:38.516 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-21T14:54:38.516 DEBUG:teuthology.orchestra.run.trial198:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-21T14:54:38.572 INFO:tasks.workunit:timeout=3h 2026-02-21T14:54:38.573 INFO:tasks.workunit:cleanup=True 2026-02-21T14:54:38.573 DEBUG:teuthology.orchestra.run.trial198:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/xxhdx1985126/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f 2026-02-21T14:54:38.635 INFO:tasks.workunit.client.0.trial198.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-21T14:54:56.826 INFO:tasks.workunit.client.0.trial198.stderr:Note: switching to 'c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f'. 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr:state without impacting any branches by switching back to a branch. 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-21T14:54:56.827 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr: git switch -c 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr:Or undo this operation with: 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr: git switch - 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr: 2026-02-21T14:54:56.828 INFO:tasks.workunit.client.0.trial198.stderr:HEAD is now at c8fda9e9b7a qa/suites/crimson-rados/thrash: crimson cpu num: 1 2026-02-21T14:54:56.832 DEBUG:teuthology.orchestra.run.trial198:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-21T14:54:56.888 INFO:tasks.workunit.client.0.trial198.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-21T14:54:56.891 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-21T14:54:56.891 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-21T14:54:56.980 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-21T14:54:57.012 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-21T14:54:57.037 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-21T14:54:57.039 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-21T14:54:57.039 INFO:tasks.workunit.client.0.trial198.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-21T14:54:57.065 INFO:tasks.workunit.client.0.trial198.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-21T14:54:57.069 DEBUG:teuthology.orchestra.run.trial198:> set -ex 2026-02-21T14:54:57.069 DEBUG:teuthology.orchestra.run.trial198:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-21T14:54:57.126 INFO:tasks.workunit:Running workunits matching cephtool/test_crimson_dump_metrics.sh on client.0... 2026-02-21T14:54:57.129 INFO:tasks.workunit:Running workunit cephtool/test_crimson_dump_metrics.sh... 2026-02-21T14:54:57.129 DEBUG:teuthology.orchestra.run.trial198:workunit test cephtool/test_crimson_dump_metrics.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh 2026-02-21T14:54:57.190 INFO:tasks.workunit.client.0.trial198.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T14:54:57.193 INFO:tasks.workunit.client.0.trial198.stderr:+ test_dump_metrics 2026-02-21T14:54:57.193 INFO:tasks.workunit.client.0.trial198.stderr:+ ceph tell osd.0 dump_metrics 2026-02-21T14:54:57.307 INFO:tasks.workunit.client.0.trial198.stderr:+ ceph tell osd.0 dump_metrics reactor_utilization 2026-02-21T14:54:57.417 INFO:tasks.workunit.client.0.trial198.stderr:++ grep -c reactor_utilization /tmp/dump_metrics_full.json 2026-02-21T14:54:57.418 INFO:tasks.workunit.client.0.trial198.stderr:+ full_count=2 2026-02-21T14:54:57.418 INFO:tasks.workunit.client.0.trial198.stderr:++ grep -c reactor_utilization /tmp/dump_metrics_reactor.json 2026-02-21T14:54:57.420 INFO:tasks.workunit.client.0.trial198.stderr:+ reactor_count=0 2026-02-21T14:54:57.420 INFO:tasks.workunit.client.0.trial198.stderr:+ '[' 2 -gt 0 ']' 2026-02-21T14:54:57.420 INFO:tasks.workunit.client.0.trial198.stderr:+ '[' 0 -gt 0 ']' 2026-02-21T14:54:57.420 INFO:tasks.workunit.client.0.trial198.stderr:+ return 1 2026-02-21T14:54:57.420 INFO:tasks.workunit.client.0.trial198.stderr:+ echo 'test_dump_metrics failed' 2026-02-21T14:54:57.420 INFO:tasks.workunit.client.0.trial198.stdout:test_dump_metrics failed 2026-02-21T14:54:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:54:57.421 INFO:tasks.workunit.client.0.trial198.stderr:+ exit 1 2026-02-21T14:54:57.422 INFO:tasks.workunit:Stopping ['cephtool/test_crimson_dump_metrics.sh'] on client.0... 2026-02-21T14:54:57.422 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-21T14:54:57.755 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_xxhdx1985126_ceph_c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f/qa/tasks/workunit.py", line 125, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_xxhdx1985126_ceph_c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f/qa/tasks/workunit.py", line 433, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on trial198 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh' 2026-02-21T14:54:57.759 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-21T14:54:57.777 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-21T14:54:57.777 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-21T14:54:57.852 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:57.860+0000 7f2fee6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:57.855 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:57.865+0000 7f2fee6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:57.972 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:57.972 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-21T14:54:57.982 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-21T14:54:56.709576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286376,"kb_used":31458060,"kb_used_data":31458060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125828316,"statfs":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.040844+0000","last_change":"2026-02-21T14:54:34.050037+0000","last_active":"2026-02-21T14:54:34.040844+0000","last_peered":"2026-02-21T14:54:34.040844+0000","last_clean":"2026-02-21T14:54:34.040844+0000","last_became_active":"2026-02-21T14:54:32.070503+0000","last_became_peered":"2026-02-21T14:54:32.070503+0000","last_unstale":"2026-02-21T14:54:34.040844+0000","last_undegraded":"2026-02-21T14:54:34.040844+0000","last_fullsized":"2026-02-21T14:54:34.040844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037776+0000","last_change":"2026-02-21T14:54:34.044270+0000","last_active":"2026-02-21T14:54:34.037776+0000","last_peered":"2026-02-21T14:54:34.037776+0000","last_clean":"2026-02-21T14:54:34.037776+0000","last_became_active":"2026-02-21T14:54:32.079015+0000","last_became_peered":"2026-02-21T14:54:32.079015+0000","last_unstale":"2026-02-21T14:54:34.037776+0000","last_undegraded":"2026-02-21T14:54:34.037776+0000","last_fullsized":"2026-02-21T14:54:34.037776+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038816+0000","last_change":"2026-02-21T14:54:34.042321+0000","last_active":"2026-02-21T14:54:34.038816+0000","last_peered":"2026-02-21T14:54:34.038816+0000","last_clean":"2026-02-21T14:54:34.038816+0000","last_became_active":"2026-02-21T14:54:32.133652+0000","last_became_peered":"2026-02-21T14:54:32.133652+0000","last_unstale":"2026-02-21T14:54:34.038816+0000","last_undegraded":"2026-02-21T14:54:34.038816+0000","last_fullsized":"2026-02-21T14:54:34.038816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041973+0000","last_change":"2026-02-21T14:54:34.050506+0000","last_active":"2026-02-21T14:54:34.041973+0000","last_peered":"2026-02-21T14:54:34.041973+0000","last_clean":"2026-02-21T14:54:34.041973+0000","last_became_active":"2026-02-21T14:54:32.111585+0000","last_became_peered":"2026-02-21T14:54:32.111585+0000","last_unstale":"2026-02-21T14:54:34.041973+0000","last_undegraded":"2026-02-21T14:54:34.041973+0000","last_fullsized":"2026-02-21T14:54:34.041973+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.042316+0000","last_change":"2026-02-21T14:54:34.050474+0000","last_active":"2026-02-21T14:54:34.042316+0000","last_peered":"2026-02-21T14:54:34.042316+0000","last_clean":"2026-02-21T14:54:34.042316+0000","last_became_active":"2026-02-21T14:54:32.074657+0000","last_became_peered":"2026-02-21T14:54:32.074657+0000","last_unstale":"2026-02-21T14:54:34.042316+0000","last_undegraded":"2026-02-21T14:54:34.042316+0000","last_fullsized":"2026-02-21T14:54:34.042316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041371+0000","last_change":"2026-02-21T14:54:34.041371+0000","last_active":"2026-02-21T14:54:34.041371+0000","last_peered":"2026-02-21T14:54:34.041371+0000","last_clean":"2026-02-21T14:54:34.041371+0000","last_became_active":"2026-02-21T14:54:32.138176+0000","last_became_peered":"2026-02-21T14:54:32.138176+0000","last_unstale":"2026-02-21T14:54:34.041371+0000","last_undegraded":"2026-02-21T14:54:34.041371+0000","last_fullsized":"2026-02-21T14:54:34.041371+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037447+0000","last_change":"2026-02-21T14:54:34.041349+0000","last_active":"2026-02-21T14:54:34.037447+0000","last_peered":"2026-02-21T14:54:34.037447+0000","last_clean":"2026-02-21T14:54:34.037447+0000","last_became_active":"2026-02-21T14:54:32.100414+0000","last_became_peered":"2026-02-21T14:54:32.100414+0000","last_unstale":"2026-02-21T14:54:34.037447+0000","last_undegraded":"2026-02-21T14:54:34.037447+0000","last_fullsized":"2026-02-21T14:54:34.037447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038019+0000","last_change":"2026-02-21T14:54:34.044647+0000","last_active":"2026-02-21T14:54:34.038019+0000","last_peered":"2026-02-21T14:54:34.038019+0000","last_clean":"2026-02-21T14:54:34.038019+0000","last_became_active":"2026-02-21T14:54:32.091454+0000","last_became_peered":"2026-02-21T14:54:32.091454+0000","last_unstale":"2026-02-21T14:54:34.038019+0000","last_undegraded":"2026-02-21T14:54:34.038019+0000","last_fullsized":"2026-02-21T14:54:34.038019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10485996,"kb_used_data":10485996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942796,"statfs":{"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-21T14:54:57.983 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-21T14:54:58.061 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:58.069+0000 7fca4a0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:58.063 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:58.074+0000 7fca4a0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:58.177 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:58.178 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-21T14:54:58.187 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-21T14:54:56.709576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286376,"kb_used":31458060,"kb_used_data":31458060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125828316,"statfs":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.040844+0000","last_change":"2026-02-21T14:54:34.050037+0000","last_active":"2026-02-21T14:54:34.040844+0000","last_peered":"2026-02-21T14:54:34.040844+0000","last_clean":"2026-02-21T14:54:34.040844+0000","last_became_active":"2026-02-21T14:54:32.070503+0000","last_became_peered":"2026-02-21T14:54:32.070503+0000","last_unstale":"2026-02-21T14:54:34.040844+0000","last_undegraded":"2026-02-21T14:54:34.040844+0000","last_fullsized":"2026-02-21T14:54:34.040844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037776+0000","last_change":"2026-02-21T14:54:34.044270+0000","last_active":"2026-02-21T14:54:34.037776+0000","last_peered":"2026-02-21T14:54:34.037776+0000","last_clean":"2026-02-21T14:54:34.037776+0000","last_became_active":"2026-02-21T14:54:32.079015+0000","last_became_peered":"2026-02-21T14:54:32.079015+0000","last_unstale":"2026-02-21T14:54:34.037776+0000","last_undegraded":"2026-02-21T14:54:34.037776+0000","last_fullsized":"2026-02-21T14:54:34.037776+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038816+0000","last_change":"2026-02-21T14:54:34.042321+0000","last_active":"2026-02-21T14:54:34.038816+0000","last_peered":"2026-02-21T14:54:34.038816+0000","last_clean":"2026-02-21T14:54:34.038816+0000","last_became_active":"2026-02-21T14:54:32.133652+0000","last_became_peered":"2026-02-21T14:54:32.133652+0000","last_unstale":"2026-02-21T14:54:34.038816+0000","last_undegraded":"2026-02-21T14:54:34.038816+0000","last_fullsized":"2026-02-21T14:54:34.038816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041973+0000","last_change":"2026-02-21T14:54:34.050506+0000","last_active":"2026-02-21T14:54:34.041973+0000","last_peered":"2026-02-21T14:54:34.041973+0000","last_clean":"2026-02-21T14:54:34.041973+0000","last_became_active":"2026-02-21T14:54:32.111585+0000","last_became_peered":"2026-02-21T14:54:32.111585+0000","last_unstale":"2026-02-21T14:54:34.041973+0000","last_undegraded":"2026-02-21T14:54:34.041973+0000","last_fullsized":"2026-02-21T14:54:34.041973+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.042316+0000","last_change":"2026-02-21T14:54:34.050474+0000","last_active":"2026-02-21T14:54:34.042316+0000","last_peered":"2026-02-21T14:54:34.042316+0000","last_clean":"2026-02-21T14:54:34.042316+0000","last_became_active":"2026-02-21T14:54:32.074657+0000","last_became_peered":"2026-02-21T14:54:32.074657+0000","last_unstale":"2026-02-21T14:54:34.042316+0000","last_undegraded":"2026-02-21T14:54:34.042316+0000","last_fullsized":"2026-02-21T14:54:34.042316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041371+0000","last_change":"2026-02-21T14:54:34.041371+0000","last_active":"2026-02-21T14:54:34.041371+0000","last_peered":"2026-02-21T14:54:34.041371+0000","last_clean":"2026-02-21T14:54:34.041371+0000","last_became_active":"2026-02-21T14:54:32.138176+0000","last_became_peered":"2026-02-21T14:54:32.138176+0000","last_unstale":"2026-02-21T14:54:34.041371+0000","last_undegraded":"2026-02-21T14:54:34.041371+0000","last_fullsized":"2026-02-21T14:54:34.041371+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037447+0000","last_change":"2026-02-21T14:54:34.041349+0000","last_active":"2026-02-21T14:54:34.037447+0000","last_peered":"2026-02-21T14:54:34.037447+0000","last_clean":"2026-02-21T14:54:34.037447+0000","last_became_active":"2026-02-21T14:54:32.100414+0000","last_became_peered":"2026-02-21T14:54:32.100414+0000","last_unstale":"2026-02-21T14:54:34.037447+0000","last_undegraded":"2026-02-21T14:54:34.037447+0000","last_fullsized":"2026-02-21T14:54:34.037447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038019+0000","last_change":"2026-02-21T14:54:34.044647+0000","last_active":"2026-02-21T14:54:34.038019+0000","last_peered":"2026-02-21T14:54:34.038019+0000","last_clean":"2026-02-21T14:54:34.038019+0000","last_became_active":"2026-02-21T14:54:32.091454+0000","last_became_peered":"2026-02-21T14:54:32.091454+0000","last_unstale":"2026-02-21T14:54:34.038019+0000","last_undegraded":"2026-02-21T14:54:34.038019+0000","last_fullsized":"2026-02-21T14:54:34.038019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10485996,"kb_used_data":10485996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942796,"statfs":{"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-21T14:54:58.189 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-21T14:54:58.189 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-21T14:54:58.265 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:58.273+0000 7fd4677a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:58.267 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:58.278+0000 7fd4677a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:58.381 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:58.381 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-21T14:54:58.390 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-21T14:54:56.709576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286376,"kb_used":31458060,"kb_used_data":31458060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125828316,"statfs":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.040844+0000","last_change":"2026-02-21T14:54:34.050037+0000","last_active":"2026-02-21T14:54:34.040844+0000","last_peered":"2026-02-21T14:54:34.040844+0000","last_clean":"2026-02-21T14:54:34.040844+0000","last_became_active":"2026-02-21T14:54:32.070503+0000","last_became_peered":"2026-02-21T14:54:32.070503+0000","last_unstale":"2026-02-21T14:54:34.040844+0000","last_undegraded":"2026-02-21T14:54:34.040844+0000","last_fullsized":"2026-02-21T14:54:34.040844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037776+0000","last_change":"2026-02-21T14:54:34.044270+0000","last_active":"2026-02-21T14:54:34.037776+0000","last_peered":"2026-02-21T14:54:34.037776+0000","last_clean":"2026-02-21T14:54:34.037776+0000","last_became_active":"2026-02-21T14:54:32.079015+0000","last_became_peered":"2026-02-21T14:54:32.079015+0000","last_unstale":"2026-02-21T14:54:34.037776+0000","last_undegraded":"2026-02-21T14:54:34.037776+0000","last_fullsized":"2026-02-21T14:54:34.037776+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038816+0000","last_change":"2026-02-21T14:54:34.042321+0000","last_active":"2026-02-21T14:54:34.038816+0000","last_peered":"2026-02-21T14:54:34.038816+0000","last_clean":"2026-02-21T14:54:34.038816+0000","last_became_active":"2026-02-21T14:54:32.133652+0000","last_became_peered":"2026-02-21T14:54:32.133652+0000","last_unstale":"2026-02-21T14:54:34.038816+0000","last_undegraded":"2026-02-21T14:54:34.038816+0000","last_fullsized":"2026-02-21T14:54:34.038816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041973+0000","last_change":"2026-02-21T14:54:34.050506+0000","last_active":"2026-02-21T14:54:34.041973+0000","last_peered":"2026-02-21T14:54:34.041973+0000","last_clean":"2026-02-21T14:54:34.041973+0000","last_became_active":"2026-02-21T14:54:32.111585+0000","last_became_peered":"2026-02-21T14:54:32.111585+0000","last_unstale":"2026-02-21T14:54:34.041973+0000","last_undegraded":"2026-02-21T14:54:34.041973+0000","last_fullsized":"2026-02-21T14:54:34.041973+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.042316+0000","last_change":"2026-02-21T14:54:34.050474+0000","last_active":"2026-02-21T14:54:34.042316+0000","last_peered":"2026-02-21T14:54:34.042316+0000","last_clean":"2026-02-21T14:54:34.042316+0000","last_became_active":"2026-02-21T14:54:32.074657+0000","last_became_peered":"2026-02-21T14:54:32.074657+0000","last_unstale":"2026-02-21T14:54:34.042316+0000","last_undegraded":"2026-02-21T14:54:34.042316+0000","last_fullsized":"2026-02-21T14:54:34.042316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041371+0000","last_change":"2026-02-21T14:54:34.041371+0000","last_active":"2026-02-21T14:54:34.041371+0000","last_peered":"2026-02-21T14:54:34.041371+0000","last_clean":"2026-02-21T14:54:34.041371+0000","last_became_active":"2026-02-21T14:54:32.138176+0000","last_became_peered":"2026-02-21T14:54:32.138176+0000","last_unstale":"2026-02-21T14:54:34.041371+0000","last_undegraded":"2026-02-21T14:54:34.041371+0000","last_fullsized":"2026-02-21T14:54:34.041371+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037447+0000","last_change":"2026-02-21T14:54:34.041349+0000","last_active":"2026-02-21T14:54:34.037447+0000","last_peered":"2026-02-21T14:54:34.037447+0000","last_clean":"2026-02-21T14:54:34.037447+0000","last_became_active":"2026-02-21T14:54:32.100414+0000","last_became_peered":"2026-02-21T14:54:32.100414+0000","last_unstale":"2026-02-21T14:54:34.037447+0000","last_undegraded":"2026-02-21T14:54:34.037447+0000","last_fullsized":"2026-02-21T14:54:34.037447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038019+0000","last_change":"2026-02-21T14:54:34.044647+0000","last_active":"2026-02-21T14:54:34.038019+0000","last_peered":"2026-02-21T14:54:34.038019+0000","last_clean":"2026-02-21T14:54:34.038019+0000","last_became_active":"2026-02-21T14:54:32.091454+0000","last_became_peered":"2026-02-21T14:54:32.091454+0000","last_unstale":"2026-02-21T14:54:34.038019+0000","last_undegraded":"2026-02-21T14:54:34.038019+0000","last_fullsized":"2026-02-21T14:54:34.038019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10485996,"kb_used_data":10485996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942796,"statfs":{"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-21T14:54:58.391 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-21T14:54:58.468 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:58.476+0000 7f9ddbf83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:58.470 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:58.481+0000 7f9ddbf83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:58.585 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:54:58.585 INFO:teuthology.orchestra.run.trial198.stdout:{"epoch":14,"fsid":"7ba0c362-e064-4d8f-9e0d-89e7be54cc27","created":"2026-02-21T14:54:14.367697+0000","modified":"2026-02-21T14:54:34.022284+0000","last_up_change":"2026-02-21T14:54:25.728036+0000","last_in_change":"2026-02-21T14:54:16.226959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-21T14:54:30.163921+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd060cd1-0fe5-49de-b464-5d41b744ed56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6804","nonce":3318032325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6805","nonce":3318032325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6811","nonce":3318032325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6814","nonce":3318032325}]},"public_addr":"10.20.193.198:6804/3318032325","cluster_addr":"10.20.193.198:6805/3318032325","heartbeat_back_addr":"10.20.193.198:6811/3318032325","heartbeat_front_addr":"10.20.193.198:6814/3318032325","state":["exists","up"]},{"osd":1,"uuid":"7573fcca-b139-4885-83fe-e9233d285f58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6803","nonce":1811593807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6802","nonce":1811593807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6809","nonce":1811593807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6806","nonce":1811593807}]},"public_addr":"10.20.193.198:6803/1811593807","cluster_addr":"10.20.193.198:6802/1811593807","heartbeat_back_addr":"10.20.193.198:6809/1811593807","heartbeat_front_addr":"10.20.193.198:6806/1811593807","state":["exists","up"]},{"osd":2,"uuid":"a0db30db-8a0b-4baa-ae29-0976e2909a6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6807","nonce":746110398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6808","nonce":746110398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6813","nonce":746110398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.198:6812","nonce":746110398}]},"public_addr":"10.20.193.198:6807/746110398","cluster_addr":"10.20.193.198:6808/746110398","heartbeat_back_addr":"10.20.193.198:6813/746110398","heartbeat_front_addr":"10.20.193.198:6812/746110398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-21T14:54:59.596 INFO:tasks.ceph:Scrubbing osd.0 2026-02-21T14:54:59.596 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-21T14:54:59.674 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:59.680+0000 7f53864cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:59.677 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:59.687+0000 7f53864cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:59.687 INFO:teuthology.orchestra.run.trial198.stdout:{ 2026-02-21T14:54:59.687 INFO:teuthology.orchestra.run.trial198.stdout: "success": "" 2026-02-21T14:54:59.687 INFO:teuthology.orchestra.run.trial198.stdout:} 2026-02-21T14:54:59.694 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-21T14:54:59.771 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:59.779+0000 7fcf7d3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:59.773 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:59.784+0000 7fcf7d3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:59.887 INFO:teuthology.orchestra.run.trial198.stderr:instructed osd(s) 0 to deep-scrub 2026-02-21T14:54:59.896 INFO:tasks.ceph:Scrubbing osd.1 2026-02-21T14:54:59.897 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-21T14:54:59.976 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:59.983+0000 7f5c52acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:59.979 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:54:59.989+0000 7f5c52acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:54:59.989 INFO:teuthology.orchestra.run.trial198.stdout:{ 2026-02-21T14:54:59.989 INFO:teuthology.orchestra.run.trial198.stdout: "success": "" 2026-02-21T14:54:59.989 INFO:teuthology.orchestra.run.trial198.stdout:} 2026-02-21T14:54:59.997 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-21T14:55:00.073 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.081+0000 7fc9de0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.076 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.086+0000 7fc9de0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.189 INFO:teuthology.orchestra.run.trial198.stderr:instructed osd(s) 1 to deep-scrub 2026-02-21T14:55:00.199 INFO:tasks.ceph:Scrubbing osd.2 2026-02-21T14:55:00.200 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-21T14:55:00.277 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.283+0000 7f7e18dad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.279 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.290+0000 7f7e18dad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.289 INFO:teuthology.orchestra.run.trial198.stdout:{ 2026-02-21T14:55:00.289 INFO:teuthology.orchestra.run.trial198.stdout: "success": "" 2026-02-21T14:55:00.290 INFO:teuthology.orchestra.run.trial198.stdout:} 2026-02-21T14:55:00.296 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-21T14:55:00.373 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.380+0000 7f0ff528b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.375 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.386+0000 7f0ff528b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.490 INFO:teuthology.orchestra.run.trial198.stderr:instructed osd(s) 2 to deep-scrub 2026-02-21T14:55:00.501 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-21T14:55:00.577 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.585+0000 7f7f0685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.579 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:00.590+0000 7f7f0685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:00.696 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:55:00.696 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-21T14:55:00.705 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-02-21T14:54:58.709913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286376,"kb_used":31458060,"kb_used_data":31458060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125828316,"statfs":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001735"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.040844+0000","last_change":"2026-02-21T14:54:34.050037+0000","last_active":"2026-02-21T14:54:34.040844+0000","last_peered":"2026-02-21T14:54:34.040844+0000","last_clean":"2026-02-21T14:54:34.040844+0000","last_became_active":"2026-02-21T14:54:32.070503+0000","last_became_peered":"2026-02-21T14:54:32.070503+0000","last_unstale":"2026-02-21T14:54:34.040844+0000","last_undegraded":"2026-02-21T14:54:34.040844+0000","last_fullsized":"2026-02-21T14:54:34.040844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037776+0000","last_change":"2026-02-21T14:54:34.044270+0000","last_active":"2026-02-21T14:54:34.037776+0000","last_peered":"2026-02-21T14:54:34.037776+0000","last_clean":"2026-02-21T14:54:34.037776+0000","last_became_active":"2026-02-21T14:54:32.079015+0000","last_became_peered":"2026-02-21T14:54:32.079015+0000","last_unstale":"2026-02-21T14:54:34.037776+0000","last_undegraded":"2026-02-21T14:54:34.037776+0000","last_fullsized":"2026-02-21T14:54:34.037776+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038816+0000","last_change":"2026-02-21T14:54:34.042321+0000","last_active":"2026-02-21T14:54:34.038816+0000","last_peered":"2026-02-21T14:54:34.038816+0000","last_clean":"2026-02-21T14:54:34.038816+0000","last_became_active":"2026-02-21T14:54:32.133652+0000","last_became_peered":"2026-02-21T14:54:32.133652+0000","last_unstale":"2026-02-21T14:54:34.038816+0000","last_undegraded":"2026-02-21T14:54:34.038816+0000","last_fullsized":"2026-02-21T14:54:34.038816+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041973+0000","last_change":"2026-02-21T14:54:34.050506+0000","last_active":"2026-02-21T14:54:34.041973+0000","last_peered":"2026-02-21T14:54:34.041973+0000","last_clean":"2026-02-21T14:54:34.041973+0000","last_became_active":"2026-02-21T14:54:32.111585+0000","last_became_peered":"2026-02-21T14:54:32.111585+0000","last_unstale":"2026-02-21T14:54:34.041973+0000","last_undegraded":"2026-02-21T14:54:34.041973+0000","last_fullsized":"2026-02-21T14:54:34.041973+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.042316+0000","last_change":"2026-02-21T14:54:34.050474+0000","last_active":"2026-02-21T14:54:34.042316+0000","last_peered":"2026-02-21T14:54:34.042316+0000","last_clean":"2026-02-21T14:54:34.042316+0000","last_became_active":"2026-02-21T14:54:32.074657+0000","last_became_peered":"2026-02-21T14:54:32.074657+0000","last_unstale":"2026-02-21T14:54:34.042316+0000","last_undegraded":"2026-02-21T14:54:34.042316+0000","last_fullsized":"2026-02-21T14:54:34.042316+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.041371+0000","last_change":"2026-02-21T14:54:34.041371+0000","last_active":"2026-02-21T14:54:34.041371+0000","last_peered":"2026-02-21T14:54:34.041371+0000","last_clean":"2026-02-21T14:54:34.041371+0000","last_became_active":"2026-02-21T14:54:32.138176+0000","last_became_peered":"2026-02-21T14:54:32.138176+0000","last_unstale":"2026-02-21T14:54:34.041371+0000","last_undegraded":"2026-02-21T14:54:34.041371+0000","last_fullsized":"2026-02-21T14:54:34.041371+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.037447+0000","last_change":"2026-02-21T14:54:34.041349+0000","last_active":"2026-02-21T14:54:34.037447+0000","last_peered":"2026-02-21T14:54:34.037447+0000","last_clean":"2026-02-21T14:54:34.037447+0000","last_became_active":"2026-02-21T14:54:32.100414+0000","last_became_peered":"2026-02-21T14:54:32.100414+0000","last_unstale":"2026-02-21T14:54:34.037447+0000","last_undegraded":"2026-02-21T14:54:34.037447+0000","last_fullsized":"2026-02-21T14:54:34.037447+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:34.038019+0000","last_change":"2026-02-21T14:54:34.044647+0000","last_active":"2026-02-21T14:54:34.038019+0000","last_peered":"2026-02-21T14:54:34.038019+0000","last_clean":"2026-02-21T14:54:34.038019+0000","last_became_active":"2026-02-21T14:54:32.091454+0000","last_became_peered":"2026-02-21T14:54:32.091454+0000","last_unstale":"2026-02-21T14:54:34.038019+0000","last_undegraded":"2026-02-21T14:54:34.038019+0000","last_fullsized":"2026-02-21T14:54:34.038019+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:31.010414+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10485996,"kb_used_data":10485996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942796,"statfs":{"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-21T14:55:00.710 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.710 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.710 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.710 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.711 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.711 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.711 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.711 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-21T14:54:31.010414+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=31, tm_wday=5, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=21, tm_hour=14, tm_min=54, tm_sec=58, tm_wday=5, tm_yday=52, tm_isdst=0) 2026-02-21T14:55:00.711 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-21T14:55:20.712 DEBUG:teuthology.orchestra.run.trial198:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-21T14:55:20.790 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:20.797+0000 7fa49feb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:20.792 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:20.802+0000 7fa49feb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:20.905 INFO:teuthology.orchestra.run.trial198.stdout: 2026-02-21T14:55:20.905 INFO:teuthology.orchestra.run.trial198.stderr:dumped all 2026-02-21T14:55:20.915 INFO:teuthology.orchestra.run.trial198.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2026-02-21T14:55:20.713852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286376,"kb_used":31458060,"kb_used_data":31458060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125828316,"statfs":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002296"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:55:00.205547+0000","last_change":"2026-02-21T14:55:00.225486+0000","last_active":"2026-02-21T14:55:00.205547+0000","last_peered":"2026-02-21T14:55:00.205547+0000","last_clean":"2026-02-21T14:55:00.205547+0000","last_became_active":"2026-02-21T14:54:32.070503+0000","last_became_peered":"2026-02-21T14:54:32.070503+0000","last_unstale":"2026-02-21T14:55:00.205547+0000","last_undegraded":"2026-02-21T14:55:00.205547+0000","last_fullsized":"2026-02-21T14:55:00.205547+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:55:00.225421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:55:00.225421+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:55:00.203919+0000","last_change":"2026-02-21T14:55:00.217903+0000","last_active":"2026-02-21T14:55:00.203919+0000","last_peered":"2026-02-21T14:55:00.203919+0000","last_clean":"2026-02-21T14:55:00.203919+0000","last_became_active":"2026-02-21T14:54:32.079015+0000","last_became_peered":"2026-02-21T14:54:32.079015+0000","last_unstale":"2026-02-21T14:55:00.203919+0000","last_undegraded":"2026-02-21T14:55:00.203919+0000","last_fullsized":"2026-02-21T14:55:00.203919+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:55:00.217850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:55:00.217850+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:55:00.502698+0000","last_change":"2026-02-21T14:55:00.516187+0000","last_active":"2026-02-21T14:55:00.502698+0000","last_peered":"2026-02-21T14:55:00.502698+0000","last_clean":"2026-02-21T14:55:00.502698+0000","last_became_active":"2026-02-21T14:54:32.133652+0000","last_became_peered":"2026-02-21T14:54:32.133652+0000","last_unstale":"2026-02-21T14:55:00.502698+0000","last_undegraded":"2026-02-21T14:55:00.502698+0000","last_fullsized":"2026-02-21T14:55:00.502698+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:55:00.516107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:55:00.516107+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:55:00.204433+0000","last_change":"2026-02-21T14:55:00.225846+0000","last_active":"2026-02-21T14:55:00.204433+0000","last_peered":"2026-02-21T14:55:00.204433+0000","last_clean":"2026-02-21T14:55:00.204433+0000","last_became_active":"2026-02-21T14:54:32.111585+0000","last_became_peered":"2026-02-21T14:54:32.111585+0000","last_unstale":"2026-02-21T14:55:00.204433+0000","last_undegraded":"2026-02-21T14:55:00.204433+0000","last_fullsized":"2026-02-21T14:55:00.204433+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:55:00.225796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:55:00.225796+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:55:00.202472+0000","last_change":"2026-02-21T14:55:00.230455+0000","last_active":"2026-02-21T14:55:00.202472+0000","last_peered":"2026-02-21T14:55:00.202472+0000","last_clean":"2026-02-21T14:55:00.202472+0000","last_became_active":"2026-02-21T14:54:32.074657+0000","last_became_peered":"2026-02-21T14:54:32.074657+0000","last_unstale":"2026-02-21T14:55:00.202472+0000","last_undegraded":"2026-02-21T14:55:00.202472+0000","last_fullsized":"2026-02-21T14:55:00.202472+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2026-02-21T14:55:00.230387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:55:00.230387+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":19,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:54:59.899122+0000","last_change":"2026-02-21T14:54:59.916852+0000","last_active":"2026-02-21T14:54:59.899122+0000","last_peered":"2026-02-21T14:54:59.899122+0000","last_clean":"2026-02-21T14:54:59.899122+0000","last_became_active":"2026-02-21T14:54:32.138176+0000","last_became_peered":"2026-02-21T14:54:32.138176+0000","last_unstale":"2026-02-21T14:54:59.899122+0000","last_undegraded":"2026-02-21T14:54:59.899122+0000","last_fullsized":"2026-02-21T14:54:59.899122+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2026-02-21T14:54:59.916764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:54:59.916764+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:55:00.502251+0000","last_change":"2026-02-21T14:55:00.516275+0000","last_active":"2026-02-21T14:55:00.502251+0000","last_peered":"2026-02-21T14:55:00.502251+0000","last_clean":"2026-02-21T14:55:00.502251+0000","last_became_active":"2026-02-21T14:54:32.100414+0000","last_became_peered":"2026-02-21T14:54:32.100414+0000","last_unstale":"2026-02-21T14:55:00.502251+0000","last_undegraded":"2026-02-21T14:55:00.502251+0000","last_fullsized":"2026-02-21T14:55:00.502251+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:55:00.516207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:55:00.516207+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-21T14:55:00.202066+0000","last_change":"2026-02-21T14:55:00.217340+0000","last_active":"2026-02-21T14:55:00.202066+0000","last_peered":"2026-02-21T14:55:00.202066+0000","last_clean":"2026-02-21T14:55:00.202066+0000","last_became_active":"2026-02-21T14:54:32.091454+0000","last_became_peered":"2026-02-21T14:54:32.091454+0000","last_unstale":"2026-02-21T14:55:00.202066+0000","last_undegraded":"2026-02-21T14:55:00.202066+0000","last_fullsized":"2026-02-21T14:55:00.202066+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-21T14:55:00.217277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-21T14:55:00.217277+0000","last_clean_scrub_stamp":"2026-02-21T14:54:31.010414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061249024,"available":128848195584,"internally_reserved":0,"allocated":32213053440,"data_stored":32213053440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10485996,"kb_used_data":10485996,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942796,"statfs":{"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428792,"kb_used":10486032,"kb_used_data":10486032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":41942760,"statfs":{"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687083008,"available":42949386240,"internally_reserved":0,"allocated":10737696768,"data_stored":10737696768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687083008,"available":42949423104,"internally_reserved":0,"allocated":10737659904,"data_stored":10737659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-21T14:55:20.917 DEBUG:teuthology.orchestra.run.trial198:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-21T14:55:20.992 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:20.997+0000 7fab4ff0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:20.994 INFO:teuthology.orchestra.run.trial198.stderr:2026-02-21T14:55:21.004+0000 7fab4ff0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-21T14:55:21.125 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-21T14:55:21.125 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-21T14:55:21.125 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-21T14:55:21.126 INFO:teuthology.orchestra.run:waiting for 300 2026-02-21T14:55:21.173 INFO:tasks.ceph.osd.0:Stopped 2026-02-21T14:55:21.174 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-21T14:55:21.174 INFO:teuthology.orchestra.run:waiting for 300 2026-02-21T14:55:21.215 INFO:tasks.ceph.osd.1:Stopped 2026-02-21T14:55:21.215 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-21T14:55:21.215 INFO:teuthology.orchestra.run:waiting for 300 2026-02-21T14:55:21.294 INFO:tasks.ceph.osd.2:Stopped 2026-02-21T14:55:21.294 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-21T14:55:21.295 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-21T14:55:21.295 INFO:teuthology.orchestra.run:waiting for 300 2026-02-21T14:55:21.367 INFO:tasks.ceph.mgr.x:Stopped 2026-02-21T14:55:21.367 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-21T14:55:21.368 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-21T14:55:21.368 INFO:teuthology.orchestra.run:waiting for 300 2026-02-21T14:55:21.417 INFO:tasks.ceph.mon.a:Stopped 2026-02-21T14:55:21.417 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-21T14:55:21.417 INFO:teuthology.orchestra.run:waiting for 300 2026-02-21T14:55:21.467 INFO:tasks.ceph.mon.b:Stopped 2026-02-21T14:55:21.467 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-21T14:55:21.467 INFO:teuthology.orchestra.run:waiting for 300 2026-02-21T14:55:21.515 INFO:tasks.ceph.mon.c:Stopped 2026-02-21T14:55:21.515 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-21T14:55:21.516 DEBUG:teuthology.orchestra.run.trial198:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v 'but it is still running' | grep -E -v 'overall HEALTH_' | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-21T14:55:21.581 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:55:21.581 DEBUG:teuthology.orchestra.run.trial198:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-21T14:55:21.653 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:55:21.653 DEBUG:teuthology.orchestra.run.trial198:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-21T14:55:21.719 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:55:21.719 DEBUG:teuthology.orchestra.run.trial198:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-21T14:55:21.787 INFO:tasks.ceph:Archiving mon data... 2026-02-21T14:55:21.788 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697/data/mon.a.tgz 2026-02-21T14:55:21.789 DEBUG:teuthology.orchestra.run.trial198:> mktemp 2026-02-21T14:55:21.844 INFO:teuthology.orchestra.run.trial198.stdout:/tmp/tmp.iTfVupBN54 2026-02-21T14:55:21.844 DEBUG:teuthology.orchestra.run.trial198:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iTfVupBN54 2026-02-21T14:55:21.924 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0666 /tmp/tmp.iTfVupBN54 2026-02-21T14:55:22.045 DEBUG:teuthology.orchestra.remote:trial198:/tmp/tmp.iTfVupBN54 is 140KB 2026-02-21T14:55:22.095 DEBUG:teuthology.orchestra.run.trial198:> rm -fr /tmp/tmp.iTfVupBN54 2026-02-21T14:55:22.111 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697/data/mon.b.tgz 2026-02-21T14:55:22.111 DEBUG:teuthology.orchestra.run.trial198:> mktemp 2026-02-21T14:55:22.168 INFO:teuthology.orchestra.run.trial198.stdout:/tmp/tmp.4djZoE5ftf 2026-02-21T14:55:22.169 DEBUG:teuthology.orchestra.run.trial198:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.4djZoE5ftf 2026-02-21T14:55:22.254 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0666 /tmp/tmp.4djZoE5ftf 2026-02-21T14:55:22.337 DEBUG:teuthology.orchestra.remote:trial198:/tmp/tmp.4djZoE5ftf is 195KB 2026-02-21T14:55:22.388 DEBUG:teuthology.orchestra.run.trial198:> rm -fr /tmp/tmp.4djZoE5ftf 2026-02-21T14:55:22.404 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697/data/mon.c.tgz 2026-02-21T14:55:22.404 DEBUG:teuthology.orchestra.run.trial198:> mktemp 2026-02-21T14:55:22.459 INFO:teuthology.orchestra.run.trial198.stdout:/tmp/tmp.t15zxFReMd 2026-02-21T14:55:22.460 DEBUG:teuthology.orchestra.run.trial198:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.t15zxFReMd 2026-02-21T14:55:22.546 DEBUG:teuthology.orchestra.run.trial198:> sudo chmod 0666 /tmp/tmp.t15zxFReMd 2026-02-21T14:55:22.628 DEBUG:teuthology.orchestra.remote:trial198:/tmp/tmp.t15zxFReMd is 195KB 2026-02-21T14:55:22.679 DEBUG:teuthology.orchestra.run.trial198:> rm -fr /tmp/tmp.t15zxFReMd 2026-02-21T14:55:22.695 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-21T14:55:22.695 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-21T14:55:22.814 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-21T14:55:22.815 INFO:tasks.ceph:Archiving crash dumps... 2026-02-21T14:55:22.817 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697/remote/trial198/crash 2026-02-21T14:55:22.818 DEBUG:teuthology.orchestra.run.trial198:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-21T14:55:22.878 INFO:tasks.ceph:Compressing logs... 2026-02-21T14:55:22.878 DEBUG:teuthology.orchestra.run.trial198:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-21T14:55:22.944 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-21T14:55:22.961 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41383.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41383.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41383.log.gz 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42751.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41730.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42167.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2026-02-21T14:55:22.962 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42751.log.gz 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43192.log 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42167.log.gz 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41878.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gzgzip 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42025.log 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41997.log 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.43192.log.gz 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41584.log 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42025.log.gz 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41997.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-21T14:55:22.963 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41997.log.gz 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42702.log 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41584.log.gz 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42604.log 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42702.log.gz 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43145.log 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42849.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42604.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.42604.log.gz -5 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42653.log 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43145.log.gz 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43045.log 2026-02-21T14:55:22.964 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42653.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42653.log.gz 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41334.log 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42996.log 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43045.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43045.log.gz 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43094.log 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41334.log.gz 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42996.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.42118.log 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42996.log.gz 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.43094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41269.log 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43094.log.gz 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42118.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38722.log 2026-02-21T14:55:22.965 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42118.log.gz 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41269.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.41269.log.gz 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph.tmp-client.admin.38722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41244.log 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38722.log.gz 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40963.log 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41244.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41244.log.gz 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40963.log: 94.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41998.log 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40963.log.gz 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41432.log 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42898.log 2026-02-21T14:55:22.966 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41998.log.gz 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41432.log.gz 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42898.log.gz 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40410.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41174.log 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41731.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41174.log.gz 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41481.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2026-02-21T14:55:22.967 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41729.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42800.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42947.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41729.log.gz 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42800.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42800.log.gz 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42503.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41128.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40749.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42503.log.gz 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41582.log 2026-02-21T14:55:22.968 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40749.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40749.log.gz 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41583.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41582.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41582.log.gz 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41850.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40528.log 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41851.log 2026-02-21T14:55:22.969 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40650.log 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40528.log.gz 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.41851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41851.log.gz 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40220.log 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40650.log.gz 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.42216.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr:/var/log/ceph/ceph-client.admin.40220.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40220.log.gz 2026-02-21T14:55:22.970 INFO:teuthology.orchestra.run.trial198.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-21T14:55:22.974 INFO:teuthology.orchestra.run.trial198.stderr: 88.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-21T14:55:22.974 INFO:teuthology.orchestra.run.trial198.stderr: 87.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-21T14:55:22.977 INFO:teuthology.orchestra.run.trial198.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-21T14:55:22.980 INFO:teuthology.orchestra.run.trial198.stderr: 88.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-21T14:55:22.987 INFO:teuthology.orchestra.run.trial198.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-21T14:55:22.999 INFO:teuthology.orchestra.run.trial198.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-21T14:55:23.000 INFO:teuthology.orchestra.run.trial198.stderr: 2026-02-21T14:55:23.000 INFO:teuthology.orchestra.run.trial198.stderr:real 0m0.064s 2026-02-21T14:55:23.001 INFO:teuthology.orchestra.run.trial198.stderr:user 0m0.222s 2026-02-21T14:55:23.001 INFO:teuthology.orchestra.run.trial198.stderr:sys 0m0.041s 2026-02-21T14:55:23.002 INFO:tasks.ceph:Archiving logs... 2026-02-21T14:55:23.002 DEBUG:teuthology.misc:Transferring archived files from trial198:/var/log/ceph to /home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697/remote/trial198/log 2026-02-21T14:55:23.003 DEBUG:teuthology.orchestra.run.trial198:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-21T14:55:23.213 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-21T14:55:23.320 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-21T14:55:23.331 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-21T14:55:23.331 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-21T14:55:23.357 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-21T14:55:23.368 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-21T14:55:23.368 DEBUG:teuthology.orchestra.run.trial198:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-21T14:55:23.411 INFO:teuthology.orchestra.run.trial198.stderr:bash: line 1: ntpq: command not found 2026-02-21T14:55:23.430 INFO:teuthology.orchestra.run.trial198.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-21T14:55:23.431 INFO:teuthology.orchestra.run.trial198.stdout:=============================================================================== 2026-02-21T14:55:23.431 INFO:teuthology.orchestra.run.trial198.stdout:^* us.gnoppix.com 2 7 377 37 -965us[ -971us] +/- 41ms 2026-02-21T14:55:23.431 INFO:teuthology.orchestra.run.trial198.stdout:^+ arm1.maxhost.io 2 7 377 37 +2056us[+2050us] +/- 54ms 2026-02-21T14:55:23.431 INFO:teuthology.orchestra.run.trial198.stdout:^+ 172-233-189-68.ip.linode> 2 7 377 36 -215us[ -215us] +/- 86ms 2026-02-21T14:55:23.431 INFO:teuthology.orchestra.run.trial198.stdout:^- 172-233-157-223.ip.linod> 2 6 377 37 -10ms[ -10ms] +/- 126ms 2026-02-21T14:55:23.432 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-21T14:55:23.447 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-21T14:55:23.447 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-21T14:55:23.462 DEBUG:teuthology.orchestra.run.trial198:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-21T14:55:23.526 DEBUG:teuthology.orchestra.run.trial198:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-21T14:55:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:55:23.553 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-21T14:55:23.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-21T14:55:23.585 INFO:teuthology.task.internal:Duration was 1685.477651 seconds 2026-02-21T14:55:23.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-21T14:55:23.602 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-21T14:55:23.602 DEBUG:teuthology.orchestra.run.trial198:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-21T14:55:23.639 INFO:teuthology.orchestra.run.trial198.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-21T14:55:23.839 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-21T14:55:23.840 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial198.front.sepia.ceph.com 2026-02-21T14:55:23.840 DEBUG:teuthology.orchestra.run.trial198:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-21T14:55:23.898 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-21T14:55:23.898 DEBUG:teuthology.orchestra.run.trial198:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-21T14:55:24.001 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-21T14:55:24.001 DEBUG:teuthology.orchestra.run.trial198:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-21T14:55:24.023 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T14:55:24.024 INFO:teuthology.orchestra.run.trial198.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T14:55:24.024 INFO:teuthology.orchestra.run.trial198.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-21T14:55:24.024 INFO:teuthology.orchestra.run.trial198.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-21T14:55:24.024 INFO:teuthology.orchestra.run.trial198.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-21T14:55:24.034 INFO:teuthology.orchestra.run.trial198.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-21T14:55:24.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-21T14:55:24.052 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-21T14:55:24.053 DEBUG:teuthology.orchestra.run.trial198:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-21T14:55:24.100 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-21T14:55:24.117 DEBUG:teuthology.orchestra.run.trial198:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-21T14:55:24.162 INFO:teuthology.orchestra.run.trial198.stdout:kernel.core_pattern = core 2026-02-21T14:55:24.181 DEBUG:teuthology.orchestra.run.trial198:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-21T14:55:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:55:24.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-21T14:55:24.253 INFO:teuthology.task.internal:Transferring archived files... 2026-02-21T14:55:24.254 DEBUG:teuthology.misc:Transferring archived files from trial198:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/xuxuehan-2026-02-21_14:23:14-crimson-rados-wip-trans-cc-opt-rbm-drop-brf-distro-debug-trial/62697/remote/trial198 2026-02-21T14:55:24.254 DEBUG:teuthology.orchestra.run.trial198:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-21T14:55:24.313 INFO:teuthology.task.internal:Removing archive directory... 2026-02-21T14:55:24.314 DEBUG:teuthology.orchestra.run.trial198:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-21T14:55:24.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-21T14:55:24.378 INFO:teuthology.task.internal:Not uploading archives. 2026-02-21T14:55:24.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-21T14:55:24.396 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-21T14:55:24.396 DEBUG:teuthology.orchestra.run.trial198:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-21T14:55:24.417 INFO:teuthology.orchestra.run.trial198.stdout: 917576 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 21 14:55 /home/ubuntu/cephtest 2026-02-21T14:55:24.417 INFO:teuthology.orchestra.run.trial198.stdout: 1442163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 21 14:54 /home/ubuntu/cephtest/mnt.0 2026-02-21T14:55:24.417 INFO:teuthology.orchestra.run.trial198.stdout: 1442164 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 21 14:54 /home/ubuntu/cephtest/mnt.0/client.0 2026-02-21T14:55:24.417 INFO:teuthology.orchestra.run.trial198.stdout: 1458048 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 14:54 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T14:55:24.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:55:24.419 INFO:teuthology.orchestra.run.trial198.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2026-02-21T14:55:24.419 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial198 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-21T14:55:24.421 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-21T14:55:24.439 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-21T14:55:24.462 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on trial198 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-21T14:55:24.468 INFO:teuthology.run:Summary data: description: crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/rbm$/{crimson_seastore_rbm} rados} duration: 1685.4776513576508 failure_reason: 'Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on trial198 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c8fda9e9b7a6ecf86711fef9f31d8fff56203e8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh''' flavor: debug owner: scheduled_xuxuehan@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-21T14:55:24.468 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-21T14:55:24.587 INFO:teuthology.run:FAIL