2026-02-25T08:01:14.822 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T08:01:14.834 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:01:14.909 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69030 branch: wip-logstoremanager description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} tasks/rbd_cls_tests} email: null first_in_suite: false flavor: debug job_id: '69030' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-logstoremanager ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: c32469671c6b0222f786dd36c34eb702f780a11d owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 869 sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: c32469671c6b0222f786dd36c34eb702f780a11d targets: trial047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOW+P+DJdirF7VUVwGcKKXyLZre3bzbnNfVfbHGu7Sej4qf6CO5gFbjPEAy6xiyqwqTT56d3h55syR00YQ1HVJY= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 - workunit: clients: client.0: - cls/test_cls_rbd.sh - cls/test_cls_lock.sh - cls/test_cls_journal.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_07:55:35 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T08:01:14.909 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa; will attempt to use it 2026-02-25T08:01:14.910 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks 2026-02-25T08:01:14.910 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T08:01:14.911 INFO:teuthology.task.internal:Checking packages... 2026-02-25T08:01:14.911 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '6aeb6975bc9a4790a399c9cc70dba9635d2a7819' 2026-02-25T08:01:14.912 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T08:01:14.912 INFO:teuthology.packaging:ref: None 2026-02-25T08:01:14.912 INFO:teuthology.packaging:tag: None 2026-02-25T08:01:14.912 INFO:teuthology.packaging:branch: wip-logstoremanager 2026-02-25T08:01:14.912 INFO:teuthology.packaging:sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:01:14.912 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-logstoremanager 2026-02-25T08:01:15.027 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5338.g6aeb6975 2026-02-25T08:01:15.028 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T08:01:15.038 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T08:01:15.038 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T08:01:15.052 INFO:teuthology.task.internal:Saving configuration 2026-02-25T08:01:15.060 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T08:01:15.071 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T08:01:15.146 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial047.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69030', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:26.746627', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOW+P+DJdirF7VUVwGcKKXyLZre3bzbnNfVfbHGu7Sej4qf6CO5gFbjPEAy6xiyqwqTT56d3h55syR00YQ1HVJY='} 2026-02-25T08:01:15.146 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T08:01:15.153 INFO:teuthology.task.internal:roles: ubuntu@trial047.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-25T08:01:15.154 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T08:01:15.244 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7abf0afa1c60>, signals=[15]) 2026-02-25T08:01:15.244 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T08:01:15.255 INFO:teuthology.task.internal:Opening connections... 2026-02-25T08:01:15.255 DEBUG:teuthology.task.internal:connecting to ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:01:15.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:01:15.322 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T08:01:15.333 DEBUG:teuthology.orchestra.run.trial047:> uname -m 2026-02-25T08:01:15.398 INFO:teuthology.orchestra.run.trial047.stdout:x86_64 2026-02-25T08:01:15.398 DEBUG:teuthology.orchestra.run.trial047:> cat /etc/os-release 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:NAME="CentOS Stream" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:VERSION="9" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:ID="centos" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:VERSION_ID="9" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:ANSI_COLOR="0;31" 2026-02-25T08:01:15.458 INFO:teuthology.orchestra.run.trial047.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:01:15.459 INFO:teuthology.orchestra.run.trial047.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:01:15.459 INFO:teuthology.orchestra.run.trial047.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:01:15.459 INFO:teuthology.orchestra.run.trial047.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:01:15.459 INFO:teuthology.orchestra.run.trial047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:01:15.459 INFO:teuthology.orchestra.run.trial047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:01:15.460 INFO:teuthology.lock.ops:Updating trial047.front.sepia.ceph.com on lock server 2026-02-25T08:01:15.553 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T08:01:15.565 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T08:01:15.578 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T08:01:15.578 DEBUG:teuthology.orchestra.run.trial047:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:01:15.593 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T08:01:15.604 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T08:01:15.604 DEBUG:teuthology.orchestra.run.trial047:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:01:15.650 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T08:01:15.754 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T08:01:15.775 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T08:01:15.775 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T08:01:15.775 DEBUG:teuthology.orchestra.run.trial047:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:01:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:01:15.790 DEBUG:teuthology.orchestra.run.trial047:> uname -r 2026-02-25T08:01:15.846 INFO:teuthology.orchestra.run.trial047.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:01:15.846 INFO:teuthology.task.kernel:Running kernel on trial047: 5.14.0-681.el9.x86_64 2026-02-25T08:01:15.846 DEBUG:teuthology.orchestra.run.trial047:> sudo yum install -y kernel 2026-02-25T08:01:17.123 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.9 MB 00:00 2026-02-25T08:01:18.016 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - AppStream 76 MB/s | 27 MB 00:00 2026-02-25T08:01:21.421 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 8.0 MB 00:00 2026-02-25T08:01:22.313 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - Extras packages 153 kB/s | 20 kB 00:00 2026-02-25T08:01:22.644 INFO:teuthology.orchestra.run.trial047.stdout:Extra Packages for Enterprise Linux 76 MB/s | 20 MB 00:00 2026-02-25T08:01:25.918 INFO:teuthology.orchestra.run.trial047.stdout:lab-extras 2.4 MB/s | 50 kB 00:00 2026-02-25T08:01:26.927 INFO:teuthology.orchestra.run.trial047.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:01:26.927 INFO:teuthology.orchestra.run.trial047.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:01:26.947 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout: Package Architecture Version Repository Size 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout:Installing dependencies: 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:01:26.948 INFO:teuthology.orchestra.run.trial047.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout:Install 4 Packages 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 89 M 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 135 M 2026-02-25T08:01:26.949 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:01:29.801 INFO:teuthology.orchestra.run.trial047.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 5.9 MB/s | 901 kB 00:00 2026-02-25T08:01:30.142 INFO:teuthology.orchestra.run.trial047.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 35 MB/s | 17 MB 00:00 2026-02-25T08:01:30.980 INFO:teuthology.orchestra.run.trial047.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 26 MB/s | 31 MB 00:01 2026-02-25T08:01:32.601 INFO:teuthology.orchestra.run.trial047.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 14 MB/s | 40 MB 00:02 2026-02-25T08:01:32.601 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:01:32.601 INFO:teuthology.orchestra.run.trial047.stdout:Total 16 MB/s | 89 MB 00:05 2026-02-25T08:01:32.794 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:01:32.997 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:01:32.997 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:01:33.381 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:01:33.381 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:01:34.491 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:01:34.779 INFO:teuthology.orchestra.run.trial047.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:01:34.801 INFO:teuthology.orchestra.run.trial047.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:01:37.375 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:01:37.408 INFO:teuthology.orchestra.run.trial047.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:01:40.447 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:01:40.453 INFO:teuthology.orchestra.run.trial047.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:01:43.502 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:01.072 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:01.094 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:01.284 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:01.284 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:02:01.284 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:02:01.284 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:02:01.341 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:02:01.342 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:02:01.403 DEBUG:teuthology.orchestra.run.trial047:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:02:01.431 INFO:teuthology.orchestra.run.trial047.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:02:01.432 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:02:01.432 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:02:01.432 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:02:01.432 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:02:01.433 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T08:02:01.433 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial047.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:02:01.433 DEBUG:teuthology.orchestra.run.trial047:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:02:01.498 INFO:teuthology.orchestra.run.trial047.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:02:01.499 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:02:01.500 DEBUG:teuthology.orchestra.run.trial047:> sudo rpm -qi grub2-tools 2026-02-25T08:02:01.568 INFO:teuthology.orchestra.run.trial047.stdout:Name : grub2-tools 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Epoch : 1 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Version : 2.06 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Release : 120.el9 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Architecture: x86_64 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Group : Unspecified 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Size : 9108611 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:License : GPLv3+ 2026-02-25T08:02:01.569 INFO:teuthology.orchestra.run.trial047.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:Packager : builder@centos.org 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:Vendor : CentOS 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:Summary : Support tools for GRUB. 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:Description : 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:02:01.570 INFO:teuthology.orchestra.run.trial047.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:02:01.571 INFO:teuthology.orchestra.run.trial047.stdout:hardware devices. 2026-02-25T08:02:01.571 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:02:01.571 INFO:teuthology.orchestra.run.trial047.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:02:01.572 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:02:01.572 INFO:teuthology.task.kernel:Updating grub on trial047 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:02:01.572 DEBUG:teuthology.orchestra.run.trial047:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:02:01.637 INFO:teuthology.orchestra.run.trial047.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:02:01.639 DEBUG:teuthology.orchestra.run.trial047:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:02:01.704 INFO:teuthology.orchestra.run.trial047.stdout:/sbin/grubby 2026-02-25T08:02:01.705 INFO:teuthology.orchestra.run.trial047.stdout:yes 2026-02-25T08:02:01.705 DEBUG:teuthology.orchestra.run.trial047:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:02:01.768 INFO:teuthology.orchestra.run.trial047.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:02:01.770 DEBUG:teuthology.orchestra.run.trial047:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:02:01.843 DEBUG:teuthology.orchestra.run.trial047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:02:02.048 INFO:teuthology.orchestra.run.trial047.stderr:Generating grub configuration file ... 2026-02-25T08:02:02.118 INFO:teuthology.orchestra.run.trial047.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:02:02.125 INFO:teuthology.orchestra.run.trial047.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:02:02.376 INFO:teuthology.orchestra.run.trial047.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:02:02.383 INFO:teuthology.orchestra.run.trial047.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:02:02.417 INFO:teuthology.orchestra.run.trial047.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:02:02.423 INFO:teuthology.orchestra.run.trial047.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:02:02.454 INFO:teuthology.orchestra.run.trial047.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:02:02.460 INFO:teuthology.orchestra.run.trial047.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:02:02.483 INFO:teuthology.orchestra.run.trial047.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:02:02.489 INFO:teuthology.orchestra.run.trial047.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:02:02.576 INFO:teuthology.orchestra.run.trial047.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:02:02.585 INFO:teuthology.orchestra.run.trial047.stderr:done 2026-02-25T08:02:02.587 DEBUG:teuthology.orchestra.run.trial047:> test -d /sys/firmware/efi 2026-02-25T08:02:02.604 DEBUG:teuthology.orchestra.run.trial047:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:02:02.669 INFO:teuthology.orchestra.run.trial047.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:02:02.669 INFO:teuthology.orchestra.run.trial047.stdout:/boot/efi/EFI/centos 2026-02-25T08:02:02.671 DEBUG:teuthology.orchestra.run.trial047:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:02:02.737 INFO:teuthology.orchestra.run.trial047.stdout:no 2026-02-25T08:02:02.738 DEBUG:teuthology.orchestra.run.trial047:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:02:02.801 INFO:teuthology.orchestra.run.trial047.stdout:yes 2026-02-25T08:02:02.802 DEBUG:teuthology.orchestra.run.trial047:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:02:02.866 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial047 2026-02-25T08:02:02.866 DEBUG:teuthology.orchestra.run.trial047:> sudo bash -c ' 2026-02-25T08:02:02.866 DEBUG:teuthology.orchestra.run.trial047:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:02:02.867 DEBUG:teuthology.orchestra.run.trial047:> ' 2026-02-25T08:02:32.909 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T08:02:32.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial047.front.sepia.ceph.com' 2026-02-25T08:02:32.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:02:47.168 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.47 2026-02-25T08:02:56.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial047.front.sepia.ceph.com' 2026-02-25T08:02:56.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:31.943 DEBUG:teuthology.orchestra.run.trial047:> true 2026-02-25T08:03:32.094 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial047.front.sepia.ceph.com' 2026-02-25T08:03:32.095 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:03:32.095 DEBUG:teuthology.orchestra.run.trial047:> uname -r 2026-02-25T08:03:32.150 INFO:teuthology.orchestra.run.trial047.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:03:32.150 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:03:32.151 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:03:32.151 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:03:33.151 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T08:03:33.152 DEBUG:teuthology.orchestra.run.trial047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:03:33.182 INFO:teuthology.orchestra.run.trial047.stdout:ttyS1 2026-02-25T08:03:33.209 DEBUG:teuthology.parallel:result is None 2026-02-25T08:03:33.209 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T08:03:33.220 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T08:03:33.221 DEBUG:teuthology.orchestra.run.trial047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:03:33.264 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T08:03:33.275 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T08:03:33.287 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T08:03:33.288 DEBUG:teuthology.orchestra.run.trial047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:03:33.326 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T08:03:33.337 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T08:03:33.337 DEBUG:teuthology.orchestra.run.trial047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:03:33.385 INFO:teuthology.orchestra.run.trial047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:03:33.394 INFO:teuthology.orchestra.run.trial047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:03:33.395 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T08:03:33.411 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T08:03:33.411 DEBUG:teuthology.orchestra.run.trial047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:03:33.460 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T08:03:33.476 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T08:03:33.476 DEBUG:teuthology.orchestra.run.trial047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:03:33.516 DEBUG:teuthology.orchestra.run.trial047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:03:33.672 DEBUG:teuthology.orchestra.run.trial047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:03:33.697 DEBUG:teuthology.orchestra.run.trial047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:03:33.837 DEBUG:teuthology.orchestra.run.trial047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:03:33.861 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:03:33.861 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:03:33.926 DEBUG:teuthology.orchestra.run.trial047:> sudo service rsyslog restart 2026-02-25T08:03:33.994 INFO:teuthology.orchestra.run.trial047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:03:34.280 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T08:03:34.292 INFO:teuthology.task.internal:Starting timer... 2026-02-25T08:03:34.292 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T08:03:34.316 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T08:03:34.334 DEBUG:teuthology.orchestra.run.trial047:> sudo service auditd rotate 2026-02-25T08:03:34.363 INFO:teuthology.orchestra.run.trial047.stdout:Rotating logs: 2026-02-25T08:03:34.365 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T08:03:34.365 DEBUG:teuthology.orchestra.run.trial047:> /usr/sbin/getenforce 2026-02-25T08:03:34.422 INFO:teuthology.orchestra.run.trial047.stdout:Permissive 2026-02-25T08:03:34.422 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial047.front.sepia.ceph.com': 'permissive'} 2026-02-25T08:03:34.423 DEBUG:teuthology.orchestra.run.trial047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:03:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:03:34.487 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T08:03:34.487 DEBUG:teuthology.orchestra.run.trial047:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:03:34.551 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T08:03:34.566 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T08:03:34.583 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T08:03:34.585 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial047.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T08:06:14.325 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial047.front.sepia.ceph.com')] 2026-02-25T08:06:14.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial047.front.sepia.ceph.com' 2026-02-25T08:06:14.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:06:14.404 DEBUG:teuthology.orchestra.run.trial047:> true 2026-02-25T08:06:14.483 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial047.front.sepia.ceph.com' 2026-02-25T08:06:14.483 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T08:06:14.497 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T08:06:14.497 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:06:14.498 DEBUG:teuthology.orchestra.run.trial047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:06:14.549 INFO:teuthology.orchestra.run.trial047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:06:14.562 INFO:teuthology.orchestra.run.trial047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:06:14.583 INFO:teuthology.orchestra.run.trial047.stderr:sudo: ntpd: command not found 2026-02-25T08:06:14.593 INFO:teuthology.orchestra.run.trial047.stdout:506 Cannot talk to daemon 2026-02-25T08:06:14.605 INFO:teuthology.orchestra.run.trial047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:06:14.616 INFO:teuthology.orchestra.run.trial047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:06:14.663 INFO:teuthology.orchestra.run.trial047.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:06:14.665 INFO:teuthology.orchestra.run.trial047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:06:14.665 INFO:teuthology.orchestra.run.trial047.stdout:=============================================================================== 2026-02-25T08:06:14.665 INFO:teuthology.orchestra.run.trial047.stdout:^? 24-111-79-186-dynamic.mi> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:06:14.665 INFO:teuthology.orchestra.run.trial047.stdout:^? 23.186.168.133 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:06:14.665 INFO:teuthology.orchestra.run.trial047.stdout:^? 172-104-28-175.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:06:14.665 INFO:teuthology.orchestra.run.trial047.stdout:^? time.gslb.gravinoc.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:06:14.666 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:06:14.677 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:06:14.677 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:06:14.677 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:06:14.677 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:06:14.681 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:06:14.681 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-25T08:06:14.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:06:14.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:06:14.808 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:06:14.808 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:06:14.937 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:06:14.937 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:06:14.937 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:06:14.963 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:06:14.964 DEBUG:teuthology.orchestra.run.trial047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:06:15.028 DEBUG:teuthology.orchestra.run.trial047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:06:15.102 DEBUG:teuthology.orchestra.run.trial047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:06:15.164 INFO:teuthology.orchestra.run.trial047.stdout:check_obsoletes = 1 2026-02-25T08:06:15.165 DEBUG:teuthology.orchestra.run.trial047:> sudo yum clean all 2026-02-25T08:06:15.312 INFO:teuthology.orchestra.run.trial047.stdout:41 files removed 2026-02-25T08:06:15.327 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-radosgw 2026-02-25T08:06:15.962 INFO:teuthology.orchestra.run.trial047.stdout:ceph packages for x86_64 192 kB/s | 94 kB 00:00 2026-02-25T08:06:16.160 INFO:teuthology.orchestra.run.trial047.stdout:ceph noarch packages 145 kB/s | 24 kB 00:00 2026-02-25T08:06:16.293 INFO:teuthology.orchestra.run.trial047.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-25T08:06:17.129 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-25T08:06:20.355 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-25T08:06:24.379 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 8.0 MB 00:01 2026-02-25T08:06:25.567 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 20 kB 00:00 2026-02-25T08:06:26.013 INFO:teuthology.orchestra.run.trial047.stdout:Extra Packages for Enterprise Linux 53 MB/s | 20 MB 00:00 2026-02-25T08:06:29.180 INFO:teuthology.orchestra.run.trial047.stdout:lab-extras 2.1 MB/s | 50 kB 00:00 2026-02-25T08:06:30.407 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repository Size 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout:Upgrading: 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout:Installing dependencies: 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:06:30.409 INFO:teuthology.orchestra.run.trial047.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:06:30.410 INFO:teuthology.orchestra.run.trial047.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:06:30.410 INFO:teuthology.orchestra.run.trial047.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:06:30.410 INFO:teuthology.orchestra.run.trial047.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:06:30.410 INFO:teuthology.orchestra.run.trial047.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout:Installing weak dependencies: 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout:Install 35 Packages 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout:Upgrade 2 Packages 2026-02-25T08:06:30.411 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:30.412 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 61 M 2026-02-25T08:06:30.412 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:06:31.788 INFO:teuthology.orchestra.run.trial047.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 4.5 MB/s | 5.1 MB 00:01 2026-02-25T08:06:31.800 INFO:teuthology.orchestra.run.trial047.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 2.1 MB/s | 25 kB 00:00 2026-02-25T08:06:31.811 INFO:teuthology.orchestra.run.trial047.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 3.1 MB/s | 37 kB 00:00 2026-02-25T08:06:31.823 INFO:teuthology.orchestra.run.trial047.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 2.2 MB/s | 26 kB 00:00 2026-02-25T08:06:31.869 INFO:teuthology.orchestra.run.trial047.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 18 MB/s | 860 kB 00:00 2026-02-25T08:06:31.887 INFO:teuthology.orchestra.run.trial047.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 15 MB/s | 263 kB 00:00 2026-02-25T08:06:32.022 INFO:teuthology.orchestra.run.trial047.stdout:(7/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9. 8.1 MB/s | 11 MB 00:01 2026-02-25T08:06:32.033 INFO:teuthology.orchestra.run.trial047.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 3.8 MB/s | 45 kB 00:00 2026-02-25T08:06:32.049 INFO:teuthology.orchestra.run.trial047.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 12 MB/s | 185 kB 00:00 2026-02-25T08:06:32.066 INFO:teuthology.orchestra.run.trial047.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 11 MB/s | 204 kB 00:00 2026-02-25T08:06:32.087 INFO:teuthology.orchestra.run.trial047.stdout:(11/37): python3-rados-20.3.0-5338.g6aeb6975.el 15 MB/s | 324 kB 00:00 2026-02-25T08:06:32.108 INFO:teuthology.orchestra.run.trial047.stdout:(12/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 14 MB/s | 306 kB 00:00 2026-02-25T08:06:32.119 INFO:teuthology.orchestra.run.trial047.stdout:(13/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_ 24 MB/s | 5.6 MB 00:00 2026-02-25T08:06:32.120 INFO:teuthology.orchestra.run.trial047.stdout:(14/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 8.2 MB/s | 100 kB 00:00 2026-02-25T08:06:32.173 INFO:teuthology.orchestra.run.trial047.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 617 kB/s | 33 kB 00:00 2026-02-25T08:06:32.201 INFO:teuthology.orchestra.run.trial047.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 3.3 MB/s | 93 kB 00:00 2026-02-25T08:06:32.311 INFO:teuthology.orchestra.run.trial047.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 946 kB/s | 104 kB 00:00 2026-02-25T08:06:32.312 INFO:teuthology.orchestra.run.trial047.stdout:(18/37): nvme-cli-2.16-1.el9.x86_64.rpm 6.0 MB/s | 1.2 MB 00:00 2026-02-25T08:06:32.355 INFO:teuthology.orchestra.run.trial047.stdout:(19/37): libnbd-1.20.3-4.el9.x86_64.rpm 3.7 MB/s | 164 kB 00:00 2026-02-25T08:06:32.376 INFO:teuthology.orchestra.run.trial047.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.1 MB/s | 45 kB 00:00 2026-02-25T08:06:32.432 INFO:teuthology.orchestra.run.trial047.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-02-25T08:06:32.432 INFO:teuthology.orchestra.run.trial047.stdout:(22/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2026-02-25T08:06:32.456 INFO:teuthology.orchestra.run.trial047.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 12 MB/s | 292 kB 00:00 2026-02-25T08:06:32.476 INFO:teuthology.orchestra.run.trial047.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.8 MB/s | 37 kB 00:00 2026-02-25T08:06:32.479 INFO:teuthology.orchestra.run.trial047.stdout:(25/37): qatlib-25.08.0-2.el9.x86_64.rpm 5.0 MB/s | 240 kB 00:00 2026-02-25T08:06:32.496 INFO:teuthology.orchestra.run.trial047.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 3.2 MB/s | 66 kB 00:00 2026-02-25T08:06:32.948 INFO:teuthology.orchestra.run.trial047.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 656 kB/s | 308 kB 00:00 2026-02-25T08:06:33.020 INFO:teuthology.orchestra.run.trial047.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 344 kB/s | 25 kB 00:00 2026-02-25T08:06:33.096 INFO:teuthology.orchestra.run.trial047.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2026-02-25T08:06:33.175 INFO:teuthology.orchestra.run.trial047.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 859 kB/s | 67 kB 00:00 2026-02-25T08:06:33.232 INFO:teuthology.orchestra.run.trial047.stdout:(31/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 8.1 MB/s | 21 MB 00:02 2026-02-25T08:06:33.237 INFO:teuthology.orchestra.run.trial047.stdout:(32/37): libarrow-9.0.0-15.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2026-02-25T08:06:33.322 INFO:teuthology.orchestra.run.trial047.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2026-02-25T08:06:33.373 INFO:teuthology.orchestra.run.trial047.stdout:(34/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2026-02-25T08:06:33.637 INFO:teuthology.orchestra.run.trial047.stdout:(35/37): re2-20211101-20.el9.x86_64.rpm 472 kB/s | 191 kB 00:00 2026-02-25T08:06:33.958 INFO:teuthology.orchestra.run.trial047.stdout:(36/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 5.2 MB/s | 3.3 MB 00:00 2026-02-25T08:06:34.165 INFO:teuthology.orchestra.run.trial047.stdout:(37/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 3.4 MB/s | 2.7 MB 00:00 2026-02-25T08:06:34.168 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:06:34.168 INFO:teuthology.orchestra.run.trial047.stdout:Total 16 MB/s | 61 MB 00:03 2026-02-25T08:06:34.266 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:06:34.286 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:06:34.286 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:06:34.534 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:06:34.534 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:06:34.824 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:06:34.969 INFO:teuthology.orchestra.run.trial047.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:06:34.974 INFO:teuthology.orchestra.run.trial047.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:06:35.025 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:06:35.026 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:06:35.039 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:06:35.041 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:06:35.042 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:06:35.071 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:06:35.073 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:06:35.081 INFO:teuthology.orchestra.run.trial047.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:06:35.084 INFO:teuthology.orchestra.run.trial047.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:06:35.087 INFO:teuthology.orchestra.run.trial047.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:06:35.091 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:06:35.102 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:06:35.115 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:06:35.116 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:06:35.138 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:06:35.141 INFO:teuthology.orchestra.run.trial047.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:06:35.171 INFO:teuthology.orchestra.run.trial047.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:06:35.175 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:06:35.181 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:06:35.183 INFO:teuthology.orchestra.run.trial047.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:06:35.187 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:06:35.302 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:06:35.303 INFO:teuthology.orchestra.run.trial047.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:06:35.330 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:06:35.332 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:06:35.340 INFO:teuthology.orchestra.run.trial047.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:06:35.595 INFO:teuthology.orchestra.run.trial047.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:06:35.596 INFO:teuthology.orchestra.run.trial047.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:06:35.612 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:06:35.616 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:06:35.622 INFO:teuthology.orchestra.run.trial047.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:06:35.624 INFO:teuthology.orchestra.run.trial047.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:06:35.625 INFO:teuthology.orchestra.run.trial047.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:06:35.643 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:06:35.758 INFO:teuthology.orchestra.run.trial047.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:06:35.760 INFO:teuthology.orchestra.run.trial047.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:06:36.042 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:06:36.043 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:06:36.043 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:36.087 INFO:teuthology.orchestra.run.trial047.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:06:36.089 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:06:37.072 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:06:37.075 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:06:37.344 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:06:37.347 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:06:37.387 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:06:37.387 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:06:37.387 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:06:37.387 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:37.391 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:06:41.972 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:06:41.972 INFO:teuthology.orchestra.run.trial047.stdout:skipping the directory /sys 2026-02-25T08:06:41.972 INFO:teuthology.orchestra.run.trial047.stdout:skipping the directory /proc 2026-02-25T08:06:41.972 INFO:teuthology.orchestra.run.trial047.stdout:skipping the directory /mnt 2026-02-25T08:06:41.972 INFO:teuthology.orchestra.run.trial047.stdout:skipping the directory /var/tmp 2026-02-25T08:06:41.972 INFO:teuthology.orchestra.run.trial047.stdout:skipping the directory /home 2026-02-25T08:06:41.973 INFO:teuthology.orchestra.run.trial047.stdout:skipping the directory /root 2026-02-25T08:06:41.973 INFO:teuthology.orchestra.run.trial047.stdout:skipping the directory /tmp 2026-02-25T08:06:41.973 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:42.482 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:06:42.506 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:06:42.506 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:06:42.506 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:06:42.506 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:06:42.506 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:06:42.506 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:42.508 INFO:teuthology.orchestra.run.trial047.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:06:42.522 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:06:42.522 INFO:teuthology.orchestra.run.trial047.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:06:42.580 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:06:42.580 INFO:teuthology.orchestra.run.trial047.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:06:42.580 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:42.869 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:06:42.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:06:42.871 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout:Upgraded: 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:06:42.939 INFO:teuthology.orchestra.run.trial047.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:42.940 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:06:42.980 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-test 2026-02-25T08:06:43.299 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:14 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:06:43.382 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repository Size 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout:Installing dependencies: 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout:Install 12 Packages 2026-02-25T08:06:43.383 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:06:43.384 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 132 M 2026-02-25T08:06:43.384 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 988 M 2026-02-25T08:06:43.384 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:06:44.086 INFO:teuthology.orchestra.run.trial047.stdout:(1/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 327 kB/s | 153 kB 00:00 2026-02-25T08:06:44.332 INFO:teuthology.orchestra.run.trial047.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 92 kB/s | 23 kB 00:00 2026-02-25T08:06:44.354 INFO:teuthology.orchestra.run.trial047.stdout:(3/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.8 MB/s | 2.1 MB 00:00 2026-02-25T08:06:44.573 INFO:teuthology.orchestra.run.trial047.stdout:(4/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 391 kB/s | 94 kB 00:00 2026-02-25T08:06:44.696 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] libasan8-15.2.1-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.2.1-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:44.923 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:45.049 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] libasan8-15.2.1-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.2.1-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:45.112 INFO:teuthology.orchestra.run.trial047.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 692 kB/s | 525 kB 00:00 2026-02-25T08:06:45.264 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:45.292 INFO:teuthology.orchestra.run.trial047.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 245 kB/s | 176 kB 00:00 2026-02-25T08:06:45.452 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:45.630 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:45.785 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:45.816 INFO:teuthology.orchestra.run.trial047.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 331 kB/s | 233 kB 00:00 2026-02-25T08:06:45.842 INFO:teuthology.orchestra.run.trial047.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 12 MB/s | 303 kB 00:00 2026-02-25T08:06:45.867 INFO:teuthology.orchestra.run.trial047.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 64 kB 00:00 2026-02-25T08:06:45.972 INFO:teuthology.orchestra.run.trial047.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:06:46.003 INFO:teuthology.orchestra.run.trial047.stdout:(10/12): protobuf-3.14.0-17.el9.x86_64.rpm 1.4 MB/s | 1.0 MB 00:00 2026-02-25T08:06:46.084 INFO:teuthology.orchestra.run.trial047.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 561 kB/s | 122 kB 00:00 2026-02-25T08:09:22.137 INFO:teuthology.orchestra.run.trial047.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 820 kB/s | 127 MB 02:38 2026-02-25T08:09:22.140 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:09:22.140 INFO:teuthology.orchestra.run.trial047.stdout:Total 849 kB/s | 132 MB 02:38 2026-02-25T08:09:22.210 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:09:22.223 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:09:22.223 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:09:22.657 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:09:22.657 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:09:23.114 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:09:23.122 INFO:teuthology.orchestra.run.trial047.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:09:23.131 INFO:teuthology.orchestra.run.trial047.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:09:23.136 INFO:teuthology.orchestra.run.trial047.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:09:23.140 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:09:23.142 INFO:teuthology.orchestra.run.trial047.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:09:23.146 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:09:23.147 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:09:23.163 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:09:23.165 INFO:teuthology.orchestra.run.trial047.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:09:23.175 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:09:23.183 INFO:teuthology.orchestra.run.trial047.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:09:23.184 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:09:30.246 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:09:30.265 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:09:31.046 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:09:31.046 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:09:31.047 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:09:31.105 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:09:31.105 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:09:31.105 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:09:31.105 INFO:teuthology.orchestra.run.trial047.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:31.105 INFO:teuthology.orchestra.run.trial047.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:09:31.105 INFO:teuthology.orchestra.run.trial047.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:09:31.106 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:09:31.144 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph 2026-02-25T08:09:31.456 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:03:02 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:09:31.543 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:09:31.545 INFO:teuthology.orchestra.run.trial047.stdout:======================================================================================= 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repository Size 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout:======================================================================================= 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout:Installing dependencies: 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:09:31.546 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:09:31.547 INFO:teuthology.orchestra.run.trial047.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:09:31.548 INFO:teuthology.orchestra.run.trial047.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:09:31.549 INFO:teuthology.orchestra.run.trial047.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:09:31.550 INFO:teuthology.orchestra.run.trial047.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:09:31.551 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:09:31.552 INFO:teuthology.orchestra.run.trial047.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout:Installing weak dependencies: 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:09:31.553 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout:======================================================================================= 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout:Install 96 Packages 2026-02-25T08:09:31.554 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:09:31.555 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 226 M 2026-02-25T08:09:31.555 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 1.5 G 2026-02-25T08:09:31.555 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:09:32.783 INFO:teuthology.orchestra.run.trial047.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 9.8 kB/s | 6.5 kB 00:00 2026-02-25T08:09:34.707 INFO:teuthology.orchestra.run.trial047.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 362 kB/s | 937 kB 00:02 2026-02-25T08:09:35.245 INFO:teuthology.orchestra.run.trial047.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 304 kB/s | 163 kB 00:00 2026-02-25T08:09:37.613 INFO:teuthology.orchestra.run.trial047.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 377 kB/s | 2.0 MB 00:05 2026-02-25T08:09:51.065 INFO:teuthology.orchestra.run.trial047.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 273 kB/s | 4.9 MB 00:18 2026-02-25T08:09:51.419 INFO:teuthology.orchestra.run.trial047.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 162 kB/s | 57 kB 00:00 2026-02-25T08:09:52.611 INFO:teuthology.orchestra.run.trial047.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 154 kB/s | 184 kB 00:01 2026-02-25T08:10:42.305 INFO:teuthology.orchestra.run.trial047.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 225 kB/s | 15 MB 01:07 2026-02-25T08:11:05.591 INFO:teuthology.orchestra.run.trial047.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 326 kB/s | 7.4 MB 00:23 2026-02-25T08:11:05.727 INFO:teuthology.orchestra.run.trial047.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 164 kB/s | 22 kB 00:00 2026-02-25T08:11:07.019 INFO:teuthology.orchestra.run.trial047.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 230 kB/s | 297 kB 00:01 2026-02-25T08:11:07.222 INFO:teuthology.orchestra.run.trial047.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 249 kB/s | 50 kB 00:00 2026-02-25T08:11:07.292 INFO:teuthology.orchestra.run.trial047.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 247 kB/s | 17 kB 00:00 2026-02-25T08:11:08.264 INFO:teuthology.orchestra.run.trial047.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 310 kB/s | 301 kB 00:00 2026-02-25T08:11:12.793 INFO:teuthology.orchestra.run.trial047.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 230 kB/s | 1.0 MB 00:04 2026-02-25T08:11:12.900 INFO:teuthology.orchestra.run.trial047.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.2 MB/s | 351 kB 00:00 2026-02-25T08:11:12.922 INFO:teuthology.orchestra.run.trial047.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.8 MB/s | 40 kB 00:00 2026-02-25T08:11:12.944 INFO:teuthology.orchestra.run.trial047.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-25T08:11:12.989 INFO:teuthology.orchestra.run.trial047.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 18 MB/s | 794 kB 00:00 2026-02-25T08:11:13.014 INFO:teuthology.orchestra.run.trial047.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.2 MB/s | 184 kB 00:00 2026-02-25T08:11:13.038 INFO:teuthology.orchestra.run.trial047.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-25T08:11:13.123 INFO:teuthology.orchestra.run.trial047.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 15 MB/s | 1.2 MB 00:00 2026-02-25T08:11:13.147 INFO:teuthology.orchestra.run.trial047.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.4 MB/s | 106 kB 00:00 2026-02-25T08:11:13.170 INFO:teuthology.orchestra.run.trial047.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.6 MB/s | 135 kB 00:00 2026-02-25T08:11:13.194 INFO:teuthology.orchestra.run.trial047.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-25T08:11:13.219 INFO:teuthology.orchestra.run.trial047.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.1 MB/s | 126 kB 00:00 2026-02-25T08:11:13.243 INFO:teuthology.orchestra.run.trial047.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.9 MB/s | 218 kB 00:00 2026-02-25T08:11:13.267 INFO:teuthology.orchestra.run.trial047.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.3 MB/s | 182 kB 00:00 2026-02-25T08:11:13.291 INFO:teuthology.orchestra.run.trial047.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-25T08:11:13.425 INFO:teuthology.orchestra.run.trial047.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 221 kB/s | 30 kB 00:00 2026-02-25T08:11:13.613 INFO:teuthology.orchestra.run.trial047.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 16 MB/s | 3.0 MB 00:00 2026-02-25T08:11:13.638 INFO:teuthology.orchestra.run.trial047.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 593 kB/s | 15 kB 00:00 2026-02-25T08:11:13.665 INFO:teuthology.orchestra.run.trial047.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 8.9 MB/s | 246 kB 00:00 2026-02-25T08:11:13.693 INFO:teuthology.orchestra.run.trial047.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 6.8 MB/s | 188 kB 00:00 2026-02-25T08:11:13.718 INFO:teuthology.orchestra.run.trial047.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 1.6 MB/s | 42 kB 00:00 2026-02-25T08:11:13.784 INFO:teuthology.orchestra.run.trial047.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 81 MB/s | 5.3 MB 00:00 2026-02-25T08:11:13.849 INFO:teuthology.orchestra.run.trial047.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 92 MB/s | 6.0 MB 00:00 2026-02-25T08:11:13.877 INFO:teuthology.orchestra.run.trial047.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 8.5 MB/s | 244 kB 00:00 2026-02-25T08:11:13.905 INFO:teuthology.orchestra.run.trial047.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 8.6 MB/s | 249 kB 00:00 2026-02-25T08:11:13.931 INFO:teuthology.orchestra.run.trial047.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 1.8 MB/s | 48 kB 00:00 2026-02-25T08:11:13.958 INFO:teuthology.orchestra.run.trial047.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 6.5 MB/s | 177 kB 00:00 2026-02-25T08:11:13.989 INFO:teuthology.orchestra.run.trial047.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 39 MB/s | 1.2 MB 00:00 2026-02-25T08:11:14.016 INFO:teuthology.orchestra.run.trial047.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.3 MB/s | 35 kB 00:00 2026-02-25T08:11:14.080 INFO:teuthology.orchestra.run.trial047.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 95 MB/s | 6.1 MB 00:00 2026-02-25T08:11:14.109 INFO:teuthology.orchestra.run.trial047.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 15 MB/s | 442 kB 00:00 2026-02-25T08:11:14.135 INFO:teuthology.orchestra.run.trial047.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 2.9 MB/s | 77 kB 00:00 2026-02-25T08:11:14.162 INFO:teuthology.orchestra.run.trial047.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 9.8 MB/s | 267 kB 00:00 2026-02-25T08:11:14.189 INFO:teuthology.orchestra.run.trial047.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 5.6 MB/s | 157 kB 00:00 2026-02-25T08:11:14.217 INFO:teuthology.orchestra.run.trial047.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 9.8 MB/s | 277 kB 00:00 2026-02-25T08:11:14.243 INFO:teuthology.orchestra.run.trial047.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.0 MB/s | 54 kB 00:00 2026-02-25T08:11:14.423 INFO:teuthology.orchestra.run.trial047.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 107 MB/s | 19 MB 00:00 2026-02-25T08:11:14.449 INFO:teuthology.orchestra.run.trial047.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.6 MB/s | 42 kB 00:00 2026-02-25T08:11:14.476 INFO:teuthology.orchestra.run.trial047.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 7.0 MB/s | 189 kB 00:00 2026-02-25T08:11:14.502 INFO:teuthology.orchestra.run.trial047.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.4 MB/s | 90 kB 00:00 2026-02-25T08:11:14.524 INFO:teuthology.orchestra.run.trial047.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 22 kB 00:00 2026-02-25T08:11:14.622 INFO:teuthology.orchestra.run.trial047.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 8.6 MB/s | 862 kB 00:00 2026-02-25T08:11:14.798 INFO:teuthology.orchestra.run.trial047.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 3.1 MB/s | 551 kB 00:00 2026-02-25T08:11:14.819 INFO:teuthology.orchestra.run.trial047.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 953 kB/s | 19 kB 00:00 2026-02-25T08:11:14.843 INFO:teuthology.orchestra.run.trial047.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2026-02-25T08:11:14.877 INFO:teuthology.orchestra.run.trial047.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 16 MB/s | 548 kB 00:00 2026-02-25T08:11:14.898 INFO:teuthology.orchestra.run.trial047.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.4 MB/s | 29 kB 00:00 2026-02-25T08:11:14.919 INFO:teuthology.orchestra.run.trial047.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.8 MB/s | 60 kB 00:00 2026-02-25T08:11:14.939 INFO:teuthology.orchestra.run.trial047.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.1 MB/s | 43 kB 00:00 2026-02-25T08:11:14.960 INFO:teuthology.orchestra.run.trial047.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 1.5 MB/s | 32 kB 00:00 2026-02-25T08:11:14.980 INFO:teuthology.orchestra.run.trial047.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 695 kB/s | 14 kB 00:00 2026-02-25T08:11:15.004 INFO:teuthology.orchestra.run.trial047.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 7.2 MB/s | 173 kB 00:00 2026-02-25T08:11:15.032 INFO:teuthology.orchestra.run.trial047.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 12 MB/s | 358 kB 00:00 2026-02-25T08:11:15.058 INFO:teuthology.orchestra.run.trial047.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 9.6 MB/s | 254 kB 00:00 2026-02-25T08:11:15.121 INFO:teuthology.orchestra.run.trial047.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 32 MB/s | 2.0 MB 00:00 2026-02-25T08:11:15.144 INFO:teuthology.orchestra.run.trial047.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 6.3 MB/s | 144 kB 00:00 2026-02-25T08:11:15.166 INFO:teuthology.orchestra.run.trial047.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 6.1 MB/s | 139 kB 00:00 2026-02-25T08:11:15.187 INFO:teuthology.orchestra.run.trial047.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.6 MB/s | 56 kB 00:00 2026-02-25T08:11:15.208 INFO:teuthology.orchestra.run.trial047.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 522 kB/s | 11 kB 00:00 2026-02-25T08:11:15.229 INFO:teuthology.orchestra.run.trial047.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 852 kB/s | 18 kB 00:00 2026-02-25T08:11:15.249 INFO:teuthology.orchestra.run.trial047.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 1.1 MB/s | 23 kB 00:00 2026-02-25T08:11:15.270 INFO:teuthology.orchestra.run.trial047.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 962 kB/s | 20 kB 00:00 2026-02-25T08:11:15.291 INFO:teuthology.orchestra.run.trial047.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 952 kB/s | 19 kB 00:00 2026-02-25T08:11:15.311 INFO:teuthology.orchestra.run.trial047.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.3 MB/s | 26 kB 00:00 2026-02-25T08:11:15.342 INFO:teuthology.orchestra.run.trial047.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 33 MB/s | 1.0 MB 00:00 2026-02-25T08:11:15.363 INFO:teuthology.orchestra.run.trial047.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 3.7 MB/s | 79 kB 00:00 2026-02-25T08:11:15.385 INFO:teuthology.orchestra.run.trial047.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.9 MB/s | 86 kB 00:00 2026-02-25T08:11:15.406 INFO:teuthology.orchestra.run.trial047.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.7 MB/s | 58 kB 00:00 2026-02-25T08:11:15.426 INFO:teuthology.orchestra.run.trial047.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 802 kB/s | 16 kB 00:00 2026-02-25T08:11:15.448 INFO:teuthology.orchestra.run.trial047.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 4.2 MB/s | 90 kB 00:00 2026-02-25T08:11:15.468 INFO:teuthology.orchestra.run.trial047.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.5 MB/s | 31 kB 00:00 2026-02-25T08:11:15.490 INFO:teuthology.orchestra.run.trial047.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 8.4 MB/s | 188 kB 00:00 2026-02-25T08:11:15.511 INFO:teuthology.orchestra.run.trial047.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 2.8 MB/s | 59 kB 00:00 2026-02-25T08:11:15.533 INFO:teuthology.orchestra.run.trial047.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 5.5 MB/s | 125 kB 00:00 2026-02-25T08:11:15.554 INFO:teuthology.orchestra.run.trial047.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.7 MB/s | 36 kB 00:00 2026-02-25T08:11:15.575 INFO:teuthology.orchestra.run.trial047.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 3.9 MB/s | 86 kB 00:00 2026-02-25T08:11:15.597 INFO:teuthology.orchestra.run.trial047.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 4.1 MB/s | 90 kB 00:00 2026-02-25T08:11:15.618 INFO:teuthology.orchestra.run.trial047.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.2 MB/s | 48 kB 00:00 2026-02-25T08:11:15.639 INFO:teuthology.orchestra.run.trial047.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 1.0 MB/s | 22 kB 00:00 2026-02-25T08:11:15.660 INFO:teuthology.orchestra.run.trial047.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 976 kB/s | 20 kB 00:00 2026-02-25T08:11:43.524 INFO:teuthology.orchestra.run.trial047.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 468 kB/s | 51 MB 01:50 2026-02-25T08:14:29.694 INFO:teuthology.orchestra.run.trial047.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 307 kB/s | 88 MB 04:52 2026-02-25T08:14:29.695 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:14:29.695 INFO:teuthology.orchestra.run.trial047.stdout:Total 775 kB/s | 226 MB 04:58 2026-02-25T08:14:30.059 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:14:30.097 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:14:30.097 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:14:30.919 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:14:30.920 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:14:31.751 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:14:31.753 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:14:31.757 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:14:31.778 INFO:teuthology.orchestra.run.trial047.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:14:31.812 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:14:31.818 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:14:31.820 INFO:teuthology.orchestra.run.trial047.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:14:31.825 INFO:teuthology.orchestra.run.trial047.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:14:31.848 INFO:teuthology.orchestra.run.trial047.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:14:31.861 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:14:31.864 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:14:31.870 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:14:31.872 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:14:31.883 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:14:31.889 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:14:31.913 INFO:teuthology.orchestra.run.trial047.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:14:31.917 INFO:teuthology.orchestra.run.trial047.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:14:31.967 INFO:teuthology.orchestra.run.trial047.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:14:31.980 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:14:31.982 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:14:31.987 INFO:teuthology.orchestra.run.trial047.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:14:31.992 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:14:31.997 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:14:32.000 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:14:32.015 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:14:32.019 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:14:32.024 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:14:32.035 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:14:32.044 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:14:32.051 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:14:32.070 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:14:32.081 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:14:32.097 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:14:32.101 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:14:32.109 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:14:32.148 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:14:32.470 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:14:32.480 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:14:32.484 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:14:32.490 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:14:32.491 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:14:32.518 INFO:teuthology.orchestra.run.trial047.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:14:32.562 INFO:teuthology.orchestra.run.trial047.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:14:32.570 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:14:32.576 INFO:teuthology.orchestra.run.trial047.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:14:32.579 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:14:32.585 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:14:32.588 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:14:32.595 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:14:32.599 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:14:32.626 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:14:32.636 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:14:32.642 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:14:32.649 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:14:32.683 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:14:32.905 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:14:32.929 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:14:32.933 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:14:32.990 INFO:teuthology.orchestra.run.trial047.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:14:32.992 INFO:teuthology.orchestra.run.trial047.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:14:33.013 INFO:teuthology.orchestra.run.trial047.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:14:33.336 INFO:teuthology.orchestra.run.trial047.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:14:33.408 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:14:34.072 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:14:34.089 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:14:34.099 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:14:34.105 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:14:34.118 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:14:34.134 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:14:34.209 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:14:34.217 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:14:34.241 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:14:34.270 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:14:34.320 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:14:34.326 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:14:34.329 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:14:34.331 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:14:34.349 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:14:34.349 INFO:teuthology.orchestra.run.trial047.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:14:34.349 INFO:teuthology.orchestra.run.trial047.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:14:34.349 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:34.357 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:14:34.382 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:14:34.382 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:14:34.382 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:34.419 INFO:teuthology.orchestra.run.trial047.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:14:35.004 INFO:teuthology.orchestra.run.trial047.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:14:40.394 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:14:40.458 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:14:40.460 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:14:40.474 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:14:40.474 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:14:40.474 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:14:40.474 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:40.485 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:14:40.508 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:14:40.508 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:14:40.508 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:14:40.508 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:14:40.508 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:14:40.508 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:40.567 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:14:40.569 INFO:teuthology.orchestra.run.trial047.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:14:40.579 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:14:40.604 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:14:40.605 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:14:43.495 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:14:43.501 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:14:43.955 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:14:43.956 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:14:43.968 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:14:43.968 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:14:44.029 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:14:44.069 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:14:44.070 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:14:44.091 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:14:44.091 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:14:44.091 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:14:44.091 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:14:44.091 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:14:44.091 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:44.102 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:14:44.313 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:14:44.314 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:14:44.334 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:14:44.334 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:14:44.335 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:14:44.335 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:14:44.335 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:14:44.335 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:44.417 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:14:44.437 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:14:44.437 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:14:44.437 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:14:44.437 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:14:44.437 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:14:44.438 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:44.438 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:14:44.444 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:14:44.466 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:14:45.896 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:14:45.897 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:14:45.898 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:14:45.899 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:14:45.900 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:14:45.901 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:14:45.902 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:14:45.903 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:14:45.904 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:14:45.964 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:14:45.964 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:45.964 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:14:45.964 INFO:teuthology.orchestra.run.trial047.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:14:45.964 INFO:teuthology.orchestra.run.trial047.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:45.965 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:14:45.966 INFO:teuthology.orchestra.run.trial047.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:14:45.967 INFO:teuthology.orchestra.run.trial047.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:14:45.968 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:14:45.969 INFO:teuthology.orchestra.run.trial047.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:14:45.970 INFO:teuthology.orchestra.run.trial047.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:14:45.971 INFO:teuthology.orchestra.run.trial047.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:14:45.972 INFO:teuthology.orchestra.run.trial047.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:14:45.972 INFO:teuthology.orchestra.run.trial047.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:14:45.972 INFO:teuthology.orchestra.run.trial047.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:14:45.972 INFO:teuthology.orchestra.run.trial047.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:14:45.972 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:45.972 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:46.031 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-base 2026-02-25T08:14:46.346 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:17 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:46.411 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:14:46.430 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:46.431 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:46.431 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:46.452 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install cephadm 2026-02-25T08:14:46.766 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:17 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:46.831 INFO:teuthology.orchestra.run.trial047.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:14:46.850 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:46.850 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:46.850 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:46.871 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:14:47.186 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:18 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:47.270 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repo Size 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:47.271 INFO:teuthology.orchestra.run.trial047.stdout:Install 1 Package 2026-02-25T08:14:47.272 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:47.272 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 149 k 2026-02-25T08:14:47.272 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 455 k 2026-02-25T08:14:47.272 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:14:47.849 INFO:teuthology.orchestra.run.trial047.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 258 kB/s | 149 kB 00:00 2026-02-25T08:14:47.850 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:14:47.850 INFO:teuthology.orchestra.run.trial047.stdout:Total 258 kB/s | 149 kB 00:00 2026-02-25T08:14:47.850 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:14:47.855 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:14:47.855 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:14:47.886 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:14:47.886 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:14:47.942 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:14:47.943 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:14:47.965 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:14:47.965 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:14:47.966 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:14:47.966 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:48.300 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:14:48.300 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:48.300 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:14:48.301 INFO:teuthology.orchestra.run.trial047.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:48.301 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:48.301 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:48.328 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr 2026-02-25T08:14:48.643 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:19 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:48.708 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:14:48.727 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:48.728 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:48.728 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:48.749 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:14:49.063 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:20 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:49.128 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:14:49.146 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:49.147 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:49.147 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:49.168 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:14:49.483 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:20 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:49.548 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:14:49.567 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:49.567 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:49.567 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:49.589 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:14:49.903 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:20 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:49.968 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:14:49.986 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:49.987 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:49.987 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:50.009 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:14:50.321 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:21 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:50.386 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:14:50.405 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:50.406 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:50.406 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:50.427 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-osd 2026-02-25T08:14:50.741 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:21 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:50.807 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:14:50.825 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:50.826 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:50.826 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:50.847 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-osd-classic 2026-02-25T08:14:51.158 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:22 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:51.224 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:14:51.242 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:51.243 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:51.243 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:51.265 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-fuse 2026-02-25T08:14:51.584 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:22 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:51.668 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repo Size 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:14:51.669 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:51.670 INFO:teuthology.orchestra.run.trial047.stdout:Install 1 Package 2026-02-25T08:14:51.670 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:51.670 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 914 k 2026-02-25T08:14:51.670 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 2.9 M 2026-02-25T08:14:51.670 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:14:55.652 INFO:teuthology.orchestra.run.trial047.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 229 kB/s | 914 kB 00:03 2026-02-25T08:14:55.654 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:14:55.654 INFO:teuthology.orchestra.run.trial047.stdout:Total 229 kB/s | 914 kB 00:03 2026-02-25T08:14:55.654 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:14:55.659 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:14:55.659 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:14:55.693 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:14:55.693 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:14:55.783 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:14:55.801 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:56.030 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:56.093 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:56.093 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:56.093 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:14:56.094 INFO:teuthology.orchestra.run.trial047.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:56.094 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:56.094 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:56.123 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-volume 2026-02-25T08:14:56.445 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:27 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:56.510 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:14:56.529 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:56.530 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:56.530 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:56.551 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install librados-devel 2026-02-25T08:14:56.862 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:27 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:56.946 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:56.947 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:56.947 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repo Size 2026-02-25T08:14:56.947 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:56.947 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout:Install 1 Package 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 124 k 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 434 k 2026-02-25T08:14:56.948 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:14:57.517 INFO:teuthology.orchestra.run.trial047.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 217 kB/s | 124 kB 00:00 2026-02-25T08:14:57.518 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:14:57.519 INFO:teuthology.orchestra.run.trial047.stdout:Total 216 kB/s | 124 kB 00:00 2026-02-25T08:14:57.519 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:14:57.523 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:14:57.523 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:14:57.554 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:14:57.554 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:14:57.609 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:14:57.627 INFO:teuthology.orchestra.run.trial047.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:57.710 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:57.773 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:57.773 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:57.773 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:14:57.773 INFO:teuthology.orchestra.run.trial047.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:57.773 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:57.773 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:57.802 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install libcephfs2 2026-02-25T08:14:58.126 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:29 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:58.190 INFO:teuthology.orchestra.run.trial047.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:14:58.209 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:58.210 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:58.210 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:58.231 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install libcephfs-devel 2026-02-25T08:14:58.549 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:29 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:58.633 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:58.634 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:58.634 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repo Size 2026-02-25T08:14:58.634 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:58.634 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:14:58.634 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout:Install 1 Package 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 37 k 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 171 k 2026-02-25T08:14:58.635 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:14:59.004 INFO:teuthology.orchestra.run.trial047.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 101 kB/s | 37 kB 00:00 2026-02-25T08:14:59.004 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:14:59.004 INFO:teuthology.orchestra.run.trial047.stdout:Total 101 kB/s | 37 kB 00:00 2026-02-25T08:14:59.004 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:14:59.007 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:14:59.007 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:14:59.009 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:14:59.009 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:14:59.034 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:14:59.052 INFO:teuthology.orchestra.run.trial047.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:59.117 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:59.178 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:14:59.179 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:59.179 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:14:59.179 INFO:teuthology.orchestra.run.trial047.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:59.179 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:14:59.179 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:59.206 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install librados2 2026-02-25T08:14:59.527 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:30 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:14:59.591 INFO:teuthology.orchestra.run.trial047.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:14:59.609 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:14:59.610 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:14:59.610 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:14:59.631 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install librbd1 2026-02-25T08:14:59.945 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:30 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:00.010 INFO:teuthology.orchestra.run.trial047.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:15:00.029 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:00.030 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:15:00.030 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:00.051 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-rados 2026-02-25T08:15:00.367 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:31 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:00.433 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:15:00.450 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:00.451 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:15:00.451 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:00.473 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-rgw 2026-02-25T08:15:00.790 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:31 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:00.855 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:15:00.873 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:00.873 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:15:00.874 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:00.895 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-cephfs 2026-02-25T08:15:01.216 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:32 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:01.281 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:15:01.299 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:01.300 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:15:01.300 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:01.321 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-rbd 2026-02-25T08:15:01.633 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:32 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:01.698 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:15:01.715 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:01.716 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:15:01.716 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:01.737 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install rbd-fuse 2026-02-25T08:15:02.053 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:33 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:02.136 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout: Package Architecture Version Repository Size 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:15:02.137 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:02.138 INFO:teuthology.orchestra.run.trial047.stdout:Install 1 Package 2026-02-25T08:15:02.138 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:02.138 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 86 k 2026-02-25T08:15:02.138 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 236 k 2026-02-25T08:15:02.138 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:15:02.798 INFO:teuthology.orchestra.run.trial047.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 131 kB/s | 86 kB 00:00 2026-02-25T08:15:02.798 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:15:02.799 INFO:teuthology.orchestra.run.trial047.stdout:Total 130 kB/s | 86 kB 00:00 2026-02-25T08:15:02.799 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:15:02.802 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:15:02.803 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:15:02.834 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:15:02.834 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:15:02.888 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:15:02.905 INFO:teuthology.orchestra.run.trial047.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:02.995 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:03.058 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:03.058 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:03.058 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:15:03.058 INFO:teuthology.orchestra.run.trial047.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:15:03.058 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:03.058 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:03.088 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install rbd-mirror 2026-02-25T08:15:03.405 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:34 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:03.487 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:03.488 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:03.488 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repo Size 2026-02-25T08:15:03.488 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout:Install 1 Package 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 2.6 M 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 11 M 2026-02-25T08:15:03.489 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:15:06.290 INFO:teuthology.orchestra.run.trial047.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 948 kB/s | 2.6 MB 00:02 2026-02-25T08:15:06.291 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:15:06.291 INFO:teuthology.orchestra.run.trial047.stdout:Total 947 kB/s | 2.6 MB 00:02 2026-02-25T08:15:06.291 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:15:06.296 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:15:06.296 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:15:06.335 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:15:06.335 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:15:06.507 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:15:06.509 INFO:teuthology.orchestra.run.trial047.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:06.531 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:06.531 INFO:teuthology.orchestra.run.trial047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:15:06.531 INFO:teuthology.orchestra.run.trial047.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:15:06.532 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:15:06.532 INFO:teuthology.orchestra.run.trial047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:15:06.532 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:06.870 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:06.871 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:06.871 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:15:06.871 INFO:teuthology.orchestra.run.trial047.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:15:06.871 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:06.871 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:06.900 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install rbd-nbd 2026-02-25T08:15:07.218 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:38 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:07.301 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:07.301 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout: Package Architecture Version Repository Size 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout:Install 1 Package 2026-02-25T08:15:07.302 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:07.303 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 166 k 2026-02-25T08:15:07.303 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 496 k 2026-02-25T08:15:07.303 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:15:08.547 INFO:teuthology.orchestra.run.trial047.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 134 kB/s | 166 kB 00:01 2026-02-25T08:15:08.548 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:15:08.548 INFO:teuthology.orchestra.run.trial047.stdout:Total 133 kB/s | 166 kB 00:01 2026-02-25T08:15:08.548 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:15:08.554 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:15:08.554 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:15:08.585 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:15:08.585 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:15:08.641 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:15:08.659 INFO:teuthology.orchestra.run.trial047.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:08.745 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:08.808 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:15:08.809 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:08.809 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:15:08.809 INFO:teuthology.orchestra.run.trial047.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:15:08.809 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:08.809 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:08.836 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:15:09.152 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:40 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:09.218 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:15:09.235 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:09.236 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:15:09.236 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:15:09.258 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:15:09.571 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:08:40 ago on Wed 25 Feb 2026 08:06:29 AM UTC. 2026-02-25T08:15:09.654 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:15:09.655 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:09.655 INFO:teuthology.orchestra.run.trial047.stdout: Package Arch Version Repo Size 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout:Installing: 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout:Installing weak dependencies: 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout:Transaction Summary 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout:================================================================================ 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout:Install 2 Packages 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:15:09.656 INFO:teuthology.orchestra.run.trial047.stdout:Total download size: 445 M 2026-02-25T08:15:09.657 INFO:teuthology.orchestra.run.trial047.stdout:Installed size: 2.7 G 2026-02-25T08:15:09.657 INFO:teuthology.orchestra.run.trial047.stdout:Downloading Packages: 2026-02-25T08:15:57.896 INFO:teuthology.orchestra.run.trial047.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 256 kB/s | 12 MB 00:48 2026-02-25T08:38:13.198 INFO:teuthology.orchestra.run.trial047.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 320 kB/s | 433 MB 23:03 2026-02-25T08:38:13.198 INFO:teuthology.orchestra.run.trial047.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:38:13.199 INFO:teuthology.orchestra.run.trial047.stdout:Total 329 kB/s | 445 MB 23:03 2026-02-25T08:38:13.199 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction check 2026-02-25T08:38:13.205 INFO:teuthology.orchestra.run.trial047.stdout:Transaction check succeeded. 2026-02-25T08:38:13.205 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction test 2026-02-25T08:38:14.582 INFO:teuthology.orchestra.run.trial047.stdout:Transaction test succeeded. 2026-02-25T08:38:14.582 INFO:teuthology.orchestra.run.trial047.stdout:Running transaction 2026-02-25T08:38:18.176 INFO:teuthology.orchestra.run.trial047.stdout: Preparing : 1/1 2026-02-25T08:38:41.266 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:38:41.284 INFO:teuthology.orchestra.run.trial047.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:38:42.148 INFO:teuthology.orchestra.run.trial047.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:38:42.149 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:38:42.212 INFO:teuthology.orchestra.run.trial047.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:38:42.212 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:38:42.212 INFO:teuthology.orchestra.run.trial047.stdout:Installed: 2026-02-25T08:38:42.212 INFO:teuthology.orchestra.run.trial047.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:38:42.212 INFO:teuthology.orchestra.run.trial047.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:38:42.212 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:38:42.212 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:38:42.289 DEBUG:teuthology.parallel:result is None 2026-02-25T08:38:42.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:38:42.459 DEBUG:teuthology.orchestra.run.trial047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:38:42.480 INFO:teuthology.orchestra.run.trial047.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:38:42.481 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:38:42.481 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:38:42.484 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T08:38:42.485 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:38:42.485 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:38:42.546 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T08:38:42.547 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:38:42.547 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:38:42.610 DEBUG:teuthology.orchestra.run.trial047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:38:42.674 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T08:38:42.674 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:38:42.674 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:38:42.738 DEBUG:teuthology.orchestra.run.trial047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:38:42.802 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T08:38:42.803 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:38:42.803 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:38:42.866 DEBUG:teuthology.orchestra.run.trial047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:38:42.930 INFO:teuthology.run_tasks:Running task pexec... 2026-02-25T08:38:42.947 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-25T08:38:42.948 DEBUG:teuthology.orchestra.run.trial047:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:38:42.972 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:38:42.972 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:38:43.023 DEBUG:teuthology.parallel:result is None 2026-02-25T08:38:43.024 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:38:43.039 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:38:43.039 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:38:43.039 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:38:43.039 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:38:43.048 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:38:43.049 INFO:teuthology.task.install:extra packages: [] 2026-02-25T08:38:43.049 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:38:43.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:38:43.219 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:38:43.219 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:38:43.603 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:38:43.603 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:38:43.603 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:38:43.627 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:38:43.627 DEBUG:teuthology.orchestra.run.trial047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:38:43.692 DEBUG:teuthology.orchestra.run.trial047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:38:43.756 DEBUG:teuthology.orchestra.run.trial047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:38:43.812 INFO:teuthology.orchestra.run.trial047.stdout:check_obsoletes = 1 2026-02-25T08:38:43.822 DEBUG:teuthology.orchestra.run.trial047:> sudo yum clean all 2026-02-25T08:38:43.981 INFO:teuthology.orchestra.run.trial047.stdout:56 files removed 2026-02-25T08:38:43.996 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-radosgw 2026-02-25T08:38:44.864 INFO:teuthology.orchestra.run.trial047.stdout:ceph packages for x86_64 132 kB/s | 94 kB 00:00 2026-02-25T08:38:45.530 INFO:teuthology.orchestra.run.trial047.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-25T08:38:46.128 INFO:teuthology.orchestra.run.trial047.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-25T08:38:47.196 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - BaseOS 8.5 MB/s | 8.9 MB 00:01 2026-02-25T08:38:50.262 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-25T08:38:53.598 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 8.0 MB 00:00 2026-02-25T08:38:54.415 INFO:teuthology.orchestra.run.trial047.stdout:CentOS Stream 9 - Extras packages 252 kB/s | 20 kB 00:00 2026-02-25T08:38:55.477 INFO:teuthology.orchestra.run.trial047.stdout:Extra Packages for Enterprise Linux 20 MB/s | 20 MB 00:00 2026-02-25T08:38:58.716 INFO:teuthology.orchestra.run.trial047.stdout:lab-extras 820 kB/s | 50 kB 00:00 2026-02-25T08:38:59.900 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:38:59.918 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:38:59.919 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:38:59.919 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:38:59.944 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-test 2026-02-25T08:39:00.260 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:00.326 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:00.343 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:00.344 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:00.344 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:00.365 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph 2026-02-25T08:39:00.681 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:00.746 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:00.765 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:00.766 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:00.766 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:00.788 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-base 2026-02-25T08:39:01.108 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:01.173 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:01.191 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:01.191 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:01.192 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:01.213 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install cephadm 2026-02-25T08:39:01.534 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:01.599 INFO:teuthology.orchestra.run.trial047.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:01.617 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:01.617 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:01.617 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:01.638 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:39:01.962 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:02.026 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:02.044 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:02.045 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:02.045 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:02.065 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr 2026-02-25T08:39:02.388 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:02.455 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:02.472 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:02.473 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:02.473 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:02.495 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:39:02.808 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:02.874 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:02.892 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:02.892 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:02.892 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:02.913 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:39:03.234 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:03.299 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:03.316 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:03.317 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:03.317 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:03.339 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:39:03.655 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:03.720 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:03.737 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:03.738 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:03.738 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:03.759 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:39:04.076 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:04.141 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:04.159 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:04.159 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:04.160 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:04.181 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-osd 2026-02-25T08:39:04.501 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:04.566 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:04.584 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:04.584 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:04.585 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:04.606 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-osd-classic 2026-02-25T08:39:04.928 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:04.995 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:05.013 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:05.014 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:05.014 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:05.035 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-fuse 2026-02-25T08:39:05.357 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:05.423 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:05.441 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:05.441 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:05.441 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:05.464 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install ceph-volume 2026-02-25T08:39:05.784 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:05.850 INFO:teuthology.orchestra.run.trial047.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:05.867 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:05.868 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:05.868 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:05.890 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install librados-devel 2026-02-25T08:39:06.209 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:06.274 INFO:teuthology.orchestra.run.trial047.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:06.292 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:06.293 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:06.293 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:06.316 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install libcephfs2 2026-02-25T08:39:06.636 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:06.701 INFO:teuthology.orchestra.run.trial047.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:06.719 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:06.719 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:06.720 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:06.741 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install libcephfs-devel 2026-02-25T08:39:07.057 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:07.123 INFO:teuthology.orchestra.run.trial047.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:07.141 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:07.141 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:07.141 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:07.164 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install librados2 2026-02-25T08:39:07.486 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:07.551 INFO:teuthology.orchestra.run.trial047.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:07.569 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:07.569 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:07.570 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:07.591 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install librbd1 2026-02-25T08:39:07.913 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:07.978 INFO:teuthology.orchestra.run.trial047.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:07.996 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:07.997 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:07.997 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:08.019 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-rados 2026-02-25T08:39:08.340 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:08.405 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:08.423 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:08.424 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:08.424 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:08.446 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-rgw 2026-02-25T08:39:08.765 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:08.831 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:08.849 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:08.850 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:08.850 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:08.871 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-cephfs 2026-02-25T08:39:09.190 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:09.256 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:09.274 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:09.275 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:09.275 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:09.297 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install python3-rbd 2026-02-25T08:39:09.619 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:09.684 INFO:teuthology.orchestra.run.trial047.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:09.702 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:09.703 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:09.703 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:09.724 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install rbd-fuse 2026-02-25T08:39:10.050 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:10.117 INFO:teuthology.orchestra.run.trial047.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:10.134 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:10.135 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:10.135 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:10.158 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install rbd-mirror 2026-02-25T08:39:10.478 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:10.543 INFO:teuthology.orchestra.run.trial047.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:10.561 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:10.562 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:10.562 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:10.583 DEBUG:teuthology.orchestra.run.trial047:> sudo yum -y install rbd-nbd 2026-02-25T08:39:10.904 INFO:teuthology.orchestra.run.trial047.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:38:58 AM UTC. 2026-02-25T08:39:10.969 INFO:teuthology.orchestra.run.trial047.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:10.987 INFO:teuthology.orchestra.run.trial047.stdout:Dependencies resolved. 2026-02-25T08:39:10.988 INFO:teuthology.orchestra.run.trial047.stdout:Nothing to do. 2026-02-25T08:39:10.988 INFO:teuthology.orchestra.run.trial047.stdout:Complete! 2026-02-25T08:39:11.013 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:11.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:39:11.165 DEBUG:teuthology.orchestra.run.trial047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:39:11.187 INFO:teuthology.orchestra.run.trial047.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:11.187 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:11.187 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:39:11.190 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-25T08:39:11.190 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T08:39:11.326 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T08:39:11.327 DEBUG:teuthology.orchestra.run.trial047:> sudo chmod 777 /var/log/ceph 2026-02-25T08:39:11.352 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T08:39:11.352 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:39:11.416 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T08:39:11.416 DEBUG:teuthology.orchestra.run.trial047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:39:11.489 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T08:39:11.490 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T08:39:11.490 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69030', 'branch': 'wip-logstoremanager', 'description': 'crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} tasks/rbd_cls_tests}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '69030', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 869, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d', 'targets': {'trial047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOW+P+DJdirF7VUVwGcKKXyLZre3bzbnNfVfbHGu7Sej4qf6CO5gFbjPEAy6xiyqwqTT56d3h55syR00YQ1HVJY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_07:55:35', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T08:39:11.491 DEBUG:teuthology.orchestra.run.trial047:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:39:11.537 DEBUG:teuthology.orchestra.run.trial047:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:39:11.600 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:11.600 DEBUG:teuthology.orchestra.run.trial047:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:39:11.657 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:39:11.657 DEBUG:teuthology.orchestra.run.trial047:> stat /dev/vg_nvme/lv_1 2026-02-25T08:39:11.712 INFO:teuthology.orchestra.run.trial047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:39:11.712 INFO:teuthology.orchestra.run.trial047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:39:11.712 INFO:teuthology.orchestra.run.trial047.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-25T08:39:11.712 INFO:teuthology.orchestra.run.trial047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:39:11.712 INFO:teuthology.orchestra.run.trial047.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:39:11.713 INFO:teuthology.orchestra.run.trial047.stdout:Access: 2026-02-25 08:15:06.685199735 +0000 2026-02-25T08:39:11.713 INFO:teuthology.orchestra.run.trial047.stdout:Modify: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:11.713 INFO:teuthology.orchestra.run.trial047.stdout:Change: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:11.713 INFO:teuthology.orchestra.run.trial047.stdout: Birth: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:11.713 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:39:11.775 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records in 2026-02-25T08:39:11.775 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records out 2026-02-25T08:39:11.775 INFO:teuthology.orchestra.run.trial047.stderr:512 bytes copied, 0.000153514 s, 3.3 MB/s 2026-02-25T08:39:11.776 DEBUG:teuthology.orchestra.run.trial047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:39:11.834 DEBUG:teuthology.orchestra.run.trial047:> stat /dev/vg_nvme/lv_2 2026-02-25T08:39:11.889 INFO:teuthology.orchestra.run.trial047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout:Device: 6h/6d Inode: 1308 Links: 1 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout:Access: 2026-02-25 08:15:06.685199735 +0000 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout:Modify: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout:Change: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:11.890 INFO:teuthology.orchestra.run.trial047.stdout: Birth: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:11.891 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:39:11.952 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records in 2026-02-25T08:39:11.952 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records out 2026-02-25T08:39:11.952 INFO:teuthology.orchestra.run.trial047.stderr:512 bytes copied, 0.000139087 s, 3.7 MB/s 2026-02-25T08:39:11.953 DEBUG:teuthology.orchestra.run.trial047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:39:12.010 DEBUG:teuthology.orchestra.run.trial047:> stat /dev/vg_nvme/lv_3 2026-02-25T08:39:12.067 INFO:teuthology.orchestra.run.trial047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:39:12.067 INFO:teuthology.orchestra.run.trial047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:39:12.067 INFO:teuthology.orchestra.run.trial047.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-25T08:39:12.067 INFO:teuthology.orchestra.run.trial047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:39:12.067 INFO:teuthology.orchestra.run.trial047.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:39:12.067 INFO:teuthology.orchestra.run.trial047.stdout:Access: 2026-02-25 08:15:06.686199697 +0000 2026-02-25T08:39:12.068 INFO:teuthology.orchestra.run.trial047.stdout:Modify: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:12.068 INFO:teuthology.orchestra.run.trial047.stdout:Change: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:12.068 INFO:teuthology.orchestra.run.trial047.stdout: Birth: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:12.068 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:39:12.130 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records in 2026-02-25T08:39:12.130 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records out 2026-02-25T08:39:12.130 INFO:teuthology.orchestra.run.trial047.stderr:512 bytes copied, 0.000133838 s, 3.8 MB/s 2026-02-25T08:39:12.131 DEBUG:teuthology.orchestra.run.trial047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:39:12.189 DEBUG:teuthology.orchestra.run.trial047:> stat /dev/vg_nvme/lv_4 2026-02-25T08:39:12.244 INFO:teuthology.orchestra.run.trial047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout:Device: 6h/6d Inode: 1307 Links: 1 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout:Access: 2026-02-25 08:15:06.686199697 +0000 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout:Modify: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout:Change: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:12.245 INFO:teuthology.orchestra.run.trial047.stdout: Birth: 2026-02-25 08:06:36.068594949 +0000 2026-02-25T08:39:12.246 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:39:12.306 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records in 2026-02-25T08:39:12.306 INFO:teuthology.orchestra.run.trial047.stderr:1+0 records out 2026-02-25T08:39:12.307 INFO:teuthology.orchestra.run.trial047.stderr:512 bytes copied, 0.000137915 s, 3.7 MB/s 2026-02-25T08:39:12.308 DEBUG:teuthology.orchestra.run.trial047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:39:12.364 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:39:12.364 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-25T08:39:12.364 INFO:tasks.ceph:Generating config... 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-25T08:39:12.367 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-25T08:39:12.368 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-25T08:39:12.369 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-25T08:39:12.369 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:39:12.370 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-25T08:39:12.370 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-25T08:39:12.370 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-25T08:39:12.370 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-25T08:39:12.370 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-25T08:39:12.370 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-25T08:39:12.370 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T08:39:12.370 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T08:39:12.370 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T08:39:12.370 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T08:39:12.445 INFO:teuthology.orchestra.run.trial047.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T08:39:12.448 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T08:39:12.531 DEBUG:teuthology.orchestra.run.trial047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:39:12.596 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.47')] 2026-02-25T08:39:12.596 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.47', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-25T08:39:12.598 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:12.598 DEBUG:teuthology.orchestra.run.trial047:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:39:12.651 DEBUG:teuthology.orchestra.run.trial047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.47 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:39:12.731 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:12.727+0000 7f7d80ad8f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:12.731 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:12.727+0000 7f7d80ad8f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:12.731 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:12.727+0000 7f7d80ad8f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:monmaptool: generated fsid 3ac28462-018f-46f5-99a5-a4e0dc86382c 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:setting min_mon_release = tentacle 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:epoch 0 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:fsid 3ac28462-018f-46f5-99a5-a4e0dc86382c 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:last_changed 2026-02-25T08:39:12.729329+0000 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:created 2026-02-25T08:39:12.729329+0000 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:min_mon_release 20 (tentacle) 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:election_strategy: 1 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:0: [v2:10.20.193.47:3300/0,v1:10.20.193.47:6789/0] mon.a 2026-02-25T08:39:12.733 INFO:teuthology.orchestra.run.trial047.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-25T08:39:12.734 DEBUG:teuthology.orchestra.run.trial047:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:39:12.789 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3ac28462-018f-46f5-99a5-a4e0dc86382c... 2026-02-25T08:39:12.790 DEBUG:teuthology.orchestra.run.trial047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:39:12.868 INFO:teuthology.orchestra.run.trial047.stdout:[global] 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: chdir = "" 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: auth supported = cephx 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: filestore xattr use omap = true 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: mon clock drift allowed = 1.000 2026-02-25T08:39:12.869 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: osd crush chooseleaf type = 0 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: auth debug = true 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: ms die on old message = true 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: ms die on bug = true 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: mon pg warn max object skew = 0 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:39:12.870 INFO:teuthology.orchestra.run.trial047.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: osd pool default size = 2 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon osd allow primary affinity = true 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon osd allow pg remap = true 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon warn on no sortbitwise = false 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon warn on too few osds = false 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:39:12.871 INFO:teuthology.orchestra.run.trial047.stdout: mon_allow_pool_size_one = true 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: osd default data pool replay window = 5 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: mon allow pool delete = true 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: mon cluster log file level = debug 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: debug asserts on shutdown = true 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: mon health detail to clog = false 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: mon host = 10.20.193.47 2026-02-25T08:39:12.872 INFO:teuthology.orchestra.run.trial047.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: ms client mode = crc 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: ms cluster mode = crc 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: ms mon client mode = crc 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: ms mon cluster mode = crc 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: ms mon service mode = crc 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: ms service mode = crc 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: fsid = 3ac28462-018f-46f5-99a5-a4e0dc86382c 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout:[osd] 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: osd journal size = 100 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: osd scrub load threshold = 5.0 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: osd scrub max interval = 600 2026-02-25T08:39:12.873 INFO:teuthology.orchestra.run.trial047.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd mclock skip benchmark = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd recover clone overlap = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd recovery max chunk = 1048576 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd debug shutdown = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd debug op order = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd debug verify stray on activate = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd debug trim objects = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd open classes on start = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: osd debug pg log writeout = true 2026-02-25T08:39:12.874 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: osd map max advance = 10 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: journal zero on create = true 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: filestore apply finisher threads = 3 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: bdev debug aio = true 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: osd debug misdirected ops = true 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: debug monc = 20 2026-02-25T08:39:12.875 INFO:teuthology.orchestra.run.trial047.stdout: crimson cpu num = 4 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: crimson osd obc lru size = 10 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug ms = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug osd = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore backref = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore cache = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore cleaner = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore device = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore epm = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore journal = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore lba = 20 2026-02-25T08:39:12.876 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore odata = 20 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore omap = 20 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore onode = 20 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore t = 20 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: debug seastore tm = 20 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: osd objectstore = seastore 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: osd shutdown pgref assert = True 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: seastore max concurrent transactions = 8 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout:[mgr] 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: debug ms = 1 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: debug mgr = 20 2026-02-25T08:39:12.877 INFO:teuthology.orchestra.run.trial047.stdout: debug mon = 20 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: debug auth = 20 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: mgr/telemetry/nag = false 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout:[mon] 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: debug ms = 1 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: debug mon = 20 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: debug paxos = 20 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: debug auth = 20 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: mon data avail warn = 5 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: mon mgr mkfs grace = 240 2026-02-25T08:39:12.878 INFO:teuthology.orchestra.run.trial047.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: mon osd reporter subtree level = osd 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: mon osd prime pg temp = true 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.879 INFO:teuthology.orchestra.run.trial047.stdout: # 1m isn't quite enough 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: mon min osdmap epochs = 50 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: mon osdmap full prune interval = 2 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: mon osdmap full prune min = 15 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: mon osdmap full prune txsize = 2 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: paxos service trim min = 10 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: osd pool default crimson = True 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:12.880 INFO:teuthology.orchestra.run.trial047.stdout:[client] 2026-02-25T08:39:12.882 INFO:teuthology.orchestra.run.trial047.stdout: rgw cache enabled = true 2026-02-25T08:39:12.883 INFO:teuthology.orchestra.run.trial047.stdout: rgw enable ops log = true 2026-02-25T08:39:12.883 INFO:teuthology.orchestra.run.trial047.stdout: rgw enable usage log = true 2026-02-25T08:39:12.883 INFO:teuthology.orchestra.run.trial047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:39:12.883 INFO:teuthology.orchestra.run.trial047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:39:12.883 INFO:teuthology.orchestra.run.trial047.stdout:[mon.a] 2026-02-25T08:39:12.885 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T08:39:12.886 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T08:39:12.975 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T08:39:12.976 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:12.976 DEBUG:teuthology.orchestra.run.trial047:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T08:39:12.993 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:12.993 DEBUG:teuthology.orchestra.run.trial047:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T08:39:13.049 INFO:tasks.ceph:Sending monmap to node ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:39:13.049 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:13.049 DEBUG:teuthology.orchestra.run.trial047:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:39:13.049 DEBUG:teuthology.orchestra.run.trial047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:39:13.122 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:13.122 DEBUG:teuthology.orchestra.run.trial047:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:39:13.179 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T08:39:13.179 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T08:39:13.179 DEBUG:teuthology.orchestra.run.trial047:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:39:13.269 INFO:teuthology.orchestra.run.trial047.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:39:13.272 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T08:39:13.272 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T08:39:13.273 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T08:39:13.315 INFO:teuthology.orchestra.run.trial047.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T08:39:13.326 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T08:39:13.326 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-25T08:39:13.327 DEBUG:teuthology.orchestra.run.trial047:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T08:39:13.391 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:39:13.391 INFO:tasks.ceph:role: osd.0 2026-02-25T08:39:13.391 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:39:13.391 DEBUG:teuthology.orchestra.run.trial047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:39:13.455 INFO:teuthology.orchestra.run.trial047.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:39:13.456 INFO:teuthology.orchestra.run.trial047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:39:14.072 INFO:teuthology.orchestra.run.trial047.stdout:Discarding blocks...Done. 2026-02-25T08:39:14.074 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial047.front.sepia.ceph.com -o noatime 2026-02-25T08:39:14.074 DEBUG:teuthology.orchestra.run.trial047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T08:39:14.143 DEBUG:teuthology.orchestra.run.trial047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T08:39:14.210 DEBUG:teuthology.orchestra.run.trial047:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T08:39:14.274 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:39:14.274 INFO:tasks.ceph:role: osd.1 2026-02-25T08:39:14.274 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:39:14.274 DEBUG:teuthology.orchestra.run.trial047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:39:14.338 INFO:teuthology.orchestra.run.trial047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:39:14.338 INFO:teuthology.orchestra.run.trial047.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:39:14.338 INFO:teuthology.orchestra.run.trial047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:39:14.338 INFO:teuthology.orchestra.run.trial047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:39:14.339 INFO:teuthology.orchestra.run.trial047.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:39:14.339 INFO:teuthology.orchestra.run.trial047.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:39:14.339 INFO:teuthology.orchestra.run.trial047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:39:14.339 INFO:teuthology.orchestra.run.trial047.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:39:14.339 INFO:teuthology.orchestra.run.trial047.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:39:14.339 INFO:teuthology.orchestra.run.trial047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:39:14.955 INFO:teuthology.orchestra.run.trial047.stdout:Discarding blocks...Done. 2026-02-25T08:39:14.957 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial047.front.sepia.ceph.com -o noatime 2026-02-25T08:39:14.957 DEBUG:teuthology.orchestra.run.trial047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T08:39:15.025 DEBUG:teuthology.orchestra.run.trial047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T08:39:15.092 DEBUG:teuthology.orchestra.run.trial047:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T08:39:15.156 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:39:15.156 INFO:tasks.ceph:role: osd.2 2026-02-25T08:39:15.156 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:39:15.157 DEBUG:teuthology.orchestra.run.trial047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-25T08:39:15.221 INFO:teuthology.orchestra.run.trial047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:39:15.221 INFO:teuthology.orchestra.run.trial047.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:39:15.221 INFO:teuthology.orchestra.run.trial047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:39:15.221 INFO:teuthology.orchestra.run.trial047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:39:15.221 INFO:teuthology.orchestra.run.trial047.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:39:15.222 INFO:teuthology.orchestra.run.trial047.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:39:15.222 INFO:teuthology.orchestra.run.trial047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:39:15.222 INFO:teuthology.orchestra.run.trial047.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:39:15.222 INFO:teuthology.orchestra.run.trial047.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:39:15.222 INFO:teuthology.orchestra.run.trial047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:39:15.829 INFO:teuthology.orchestra.run.trial047.stdout:Discarding blocks...Done. 2026-02-25T08:39:15.831 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial047.front.sepia.ceph.com -o noatime 2026-02-25T08:39:15.831 DEBUG:teuthology.orchestra.run.trial047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-25T08:39:15.898 DEBUG:teuthology.orchestra.run.trial047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T08:39:15.965 DEBUG:teuthology.orchestra.run.trial047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:39:16.114 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:39:16.170 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:16,168 seastar - Reactor backend: linux-aio 2026-02-25T08:39:16.183 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:16,181 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:16.183 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:16,181 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:16.185 INFO:teuthology.orchestra.run.trial047.stderr:==39739==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:39:16.240 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:16,237 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:39:16.240 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:16,237 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:39:16.240 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:16,237 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:39:16.240 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:16,237 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:39:16.240 INFO:teuthology.orchestra.run.trial047.stderr:DEBUG 2026-02-25 08:39:16,238 [shard 0:main] seastar - reactor::drain 2026-02-25T08:39:18.309 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:18.310 INFO:teuthology.orchestra.run.trial047.stderr:Suppressions used: 2026-02-25T08:39:18.310 INFO:teuthology.orchestra.run.trial047.stderr: count bytes template 2026-02-25T08:39:18.310 INFO:teuthology.orchestra.run.trial047.stderr: 1 8 MallocExtension::Register 2026-02-25T08:39:18.310 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:18.310 INFO:teuthology.orchestra.run.trial047.stderr: 2026-02-25T08:39:18.330 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:39:18.330 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:39:18.373 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:39:18.373 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:39:18.380 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:18,378 seastar - Reactor backend: linux-aio 2026-02-25T08:39:18.394 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:18,392 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:18.394 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:18,392 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:18.398 INFO:teuthology.orchestra.run.trial047.stderr:==39737==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:39:18.399 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:18,397 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:39:18.461 INFO:teuthology.orchestra.run.trial047.stderr:DEBUG 2026-02-25 08:39:18,459 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:39:18.542 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129787 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x10a2823 0xf47938 0x9f044e9 0x9f04ad6 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.542 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10a2823 0xf47938 0x9f044e9 0x9f04ad6 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.542 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.575 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e21fef 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:18.618 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 66 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc589 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x782d17a 0x9558c65 0x959e7f8 0x962a7bc 0x90f061c 0x90f0af2 0x916ab89 0x7a59cdd 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.618 INFO:teuthology.orchestra.run.trial047.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa1691052 0xffffffffa2350451 0xffffffffa2400ba2 2026-02-25T08:39:18.645 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a1526e 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:18.681 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.682 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.685 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69079 0x7d56d00 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.805 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b5 0x11380c8 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.806 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1588f 0x10a2823 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.806 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9d73989 0x9d73a9c 0x9d73b78 0x9d73df4 0x9d73ead 0x9d7439b 0x9d7479f 0x9d74a5c 0x9f165e1 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.888 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521e 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:18.903 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc657f /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.903 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:18.904 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:21.072 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:21.073 INFO:teuthology.orchestra.run.trial047.stderr:Suppressions used: 2026-02-25T08:39:21.073 INFO:teuthology.orchestra.run.trial047.stderr: count bytes template 2026-02-25T08:39:21.073 INFO:teuthology.orchestra.run.trial047.stderr: 1 8 MallocExtension::Register 2026-02-25T08:39:21.073 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:21.073 INFO:teuthology.orchestra.run.trial047.stderr: 2026-02-25T08:39:21.109 DEBUG:teuthology.orchestra.run.trial047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T08:39:21.134 DEBUG:teuthology.orchestra.run.trial047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:39:21.287 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:39:21.344 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:21,342 seastar - Reactor backend: linux-aio 2026-02-25T08:39:21.353 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:21,351 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:21.353 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:21,351 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:21.355 INFO:teuthology.orchestra.run.trial047.stderr:==39805==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:39:21.411 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:21,408 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:39:21.411 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:21,408 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:39:21.411 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:21,408 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:39:21.411 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:21,408 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:39:21.411 INFO:teuthology.orchestra.run.trial047.stderr:DEBUG 2026-02-25 08:39:21,409 [shard 0:main] seastar - reactor::drain 2026-02-25T08:39:23.474 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:23.475 INFO:teuthology.orchestra.run.trial047.stderr:Suppressions used: 2026-02-25T08:39:23.475 INFO:teuthology.orchestra.run.trial047.stderr: count bytes template 2026-02-25T08:39:23.475 INFO:teuthology.orchestra.run.trial047.stderr: 1 8 MallocExtension::Register 2026-02-25T08:39:23.475 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:23.475 INFO:teuthology.orchestra.run.trial047.stderr: 2026-02-25T08:39:23.495 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:39:23.496 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:39:23.541 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:39:23.541 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:39:23.548 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:23,546 seastar - Reactor backend: linux-aio 2026-02-25T08:39:23.562 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:23,560 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:23.562 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:23,560 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:23.566 INFO:teuthology.orchestra.run.trial047.stderr:==39803==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:39:23.566 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:23,564 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:39:23.626 INFO:teuthology.orchestra.run.trial047.stderr:DEBUG 2026-02-25 08:39:23,624 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:39:23.708 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcc27ed 0x1138042 0xf4792d 0x9f044e9 0x9f04ad6 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.708 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf61eb 0x9edd18e 0x9edd2bc 0x9f0f6c7 0x9f0fe6a 0x9f1a337 0x78b69db 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.708 INFO:teuthology.orchestra.run.trial047.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa19ea09b 0xffffffffa19cc9fd 0xffffffffa19d4bab 0xffffffffa19d500b 0xffffffffa16910d2 0xffffffffa2350451 0xffffffffa2400ba2 2026-02-25T08:39:23.708 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xbac584 0x9f0f878 0x9f0fe6a 0x9f1a337 0x78b69db 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.740 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15f26 0xbac435 0xc862f3 0xc86588 0xc8c9ab 0xc8ceea 0xa7a7d5a 0xc7e26a 0xa7f3431 0xa7f373d 0xa7fcfb6 0x95d7664 0x962a4e7 0x90f061c 0x90f0af2 0x916a40e 0x7a59cdd 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:23.807 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a15245 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:23.847 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x7d56ce2 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.848 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1523f 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.848 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69801 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.972 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x157c7 0x9e21e70 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.973 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x755a777 0x725c0b0 0x711b842 0x7ba1535 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.973 INFO:teuthology.orchestra.run.trial047.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa1675c69 0xffffffffa1820157 0xffffffffa18203b0 0xffffffffa169525e 0xffffffffa19e7269 0xffffffffa19ca7d5 0xffffffffa19d4bab 0xffffffffa19d500b 0xffffffffa16910d2 0xffffffffa2350451 0xffffffffa2400ba2 2026-02-25T08:39:23.973 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x755a777 0x725c0b0 0x711b842 0x7ba1535 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:23.973 INFO:teuthology.orchestra.run.trial047.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa1675c69 0xffffffffa1820157 0xffffffffa18203b0 0xffffffffa169525e 0xffffffffa19e7269 0xffffffffa19ca7d5 0xffffffffa19d4bab 0xffffffffa19d500b 0xffffffffa16910d2 0xffffffffa2350451 0xffffffffa2400ba2 2026-02-25T08:39:24.055 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbff /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:24.071 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:24.071 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:24.071 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69624 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:26.262 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:26.262 INFO:teuthology.orchestra.run.trial047.stderr:Suppressions used: 2026-02-25T08:39:26.262 INFO:teuthology.orchestra.run.trial047.stderr: count bytes template 2026-02-25T08:39:26.263 INFO:teuthology.orchestra.run.trial047.stderr: 1 8 MallocExtension::Register 2026-02-25T08:39:26.263 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:26.263 INFO:teuthology.orchestra.run.trial047.stderr: 2026-02-25T08:39:26.299 DEBUG:teuthology.orchestra.run.trial047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T08:39:26.325 DEBUG:teuthology.orchestra.run.trial047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:39:26.476 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:39:26.537 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:26,534 seastar - Reactor backend: linux-aio 2026-02-25T08:39:26.547 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:26,545 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:26.547 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:26,545 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:26.549 INFO:teuthology.orchestra.run.trial047.stderr:==39872==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:39:26.605 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:26,602 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:39:26.605 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:26,602 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:39:26.605 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:26,602 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:39:26.605 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:26,602 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:39:26.606 INFO:teuthology.orchestra.run.trial047.stderr:DEBUG 2026-02-25 08:39:26,603 [shard 0:main] seastar - reactor::drain 2026-02-25T08:39:28.617 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:28.618 INFO:teuthology.orchestra.run.trial047.stderr:Suppressions used: 2026-02-25T08:39:28.618 INFO:teuthology.orchestra.run.trial047.stderr: count bytes template 2026-02-25T08:39:28.618 INFO:teuthology.orchestra.run.trial047.stderr: 1 8 MallocExtension::Register 2026-02-25T08:39:28.618 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:28.618 INFO:teuthology.orchestra.run.trial047.stderr: 2026-02-25T08:39:28.638 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:39:28.638 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:39:28.684 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:39:28.684 INFO:teuthology.orchestra.run.trial047.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:39:28.691 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:28,688 seastar - Reactor backend: linux-aio 2026-02-25T08:39:28.703 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:28,701 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:28.703 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:28,701 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:39:28.707 INFO:teuthology.orchestra.run.trial047.stderr:==39870==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:39:28.707 INFO:teuthology.orchestra.run.trial047.stderr:INFO 2026-02-25 08:39:28,705 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:39:28.765 INFO:teuthology.orchestra.run.trial047.stderr:DEBUG 2026-02-25 08:39:28,763 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:39:28.845 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10069f /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10a2823 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e22e1c 0x9e2ae9f 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:28.845 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129747 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x11380c8 0xf4792d 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9f09a9d 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:28.845 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x11380c8 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:28.878 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0xff171 /lib64/libasan.so.8+0x470c3 /lib64/libstdc++.so.6+0xd0f6d /lib64/libstdc++.so.6+0x1138d1 /lib64/libstdc++.so.6+0x113d17 /lib64/libstdc++.so.6+0x111a9c /lib64/libstdc++.so.6+0x139f62 0xa7fd12c 0x95d7664 0x962a4e7 0x90f061c 0x90f0af2 0x916ab89 0x7a59cdd 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:28.944 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:28.984 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a690b3 0x7d56d00 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:28.984 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:28.984 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521e 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:29.116 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9f19da5 0x78b69db 0x7ba11e3 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:29.116 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xb7fb36 0x10a2729 0xf47938 0x9e26106 0x9e263b3 0x9f138a5 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:29.116 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9f0af45 0x9f0f82c 0x9f0fe6a 0x9f1a337 0x78b69db 0x7ba11e3 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:29.199 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:39:29.214 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:29.214 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:29.214 INFO:teuthology.orchestra.run.trial047.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:39:31.349 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:31.350 INFO:teuthology.orchestra.run.trial047.stderr:Suppressions used: 2026-02-25T08:39:31.350 INFO:teuthology.orchestra.run.trial047.stderr: count bytes template 2026-02-25T08:39:31.350 INFO:teuthology.orchestra.run.trial047.stderr: 1 8 MallocExtension::Register 2026-02-25T08:39:31.350 INFO:teuthology.orchestra.run.trial047.stderr:----------------------------------------------------- 2026-02-25T08:39:31.350 INFO:teuthology.orchestra.run.trial047.stderr: 2026-02-25T08:39:31.387 DEBUG:teuthology.orchestra.run.trial047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T08:39:31.412 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T08:39:31.413 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:31.413 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T08:39:31.477 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:31.477 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T08:39:31.542 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:31.542 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T08:39:31.606 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:31.606 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T08:39:31.670 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:31.670 DEBUG:teuthology.orchestra.run.trial047:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T08:39:31.725 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T08:39:31.726 DEBUG:teuthology.orchestra.run.trial047:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T08:39:31.787 INFO:teuthology.orchestra.run.trial047.stdout:[mgr.x] 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout: key = AQAxtZ5pYVjuDxAA2dNiCErqVNl+OKfA/9Z/rg== 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout:[osd.0] 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout: key = AQA2tZ5p25SfGxAAL13PFXEvyQxJfBLqg0BLfg== 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout:[osd.1] 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout: key = AQA7tZ5pFTpvJRAAxszhjXcjnOANKJkgdlX09g== 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout:[osd.2] 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout: key = AQBAtZ5pLcS0LRAAUGZfzupIwwAWpEx/qyqg2g== 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout:[client.0] 2026-02-25T08:39:31.788 INFO:teuthology.orchestra.run.trial047.stdout: key = AQAxtZ5po0moEhAA61UXCYhHaleQKZ/Eu/nuSQ== 2026-02-25T08:39:31.790 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T08:39:31.879 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:39:31.927 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:39:31.975 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:39:32.026 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:39:32.076 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T08:39:32.077 DEBUG:teuthology.orchestra.run.trial047:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T08:39:32.102 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:39:32.187 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.184+0000 7fbf20288c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.187 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.184+0000 7fbf20288c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.199 DEBUG:teuthology.orchestra.run.trial047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T08:39:32.223 DEBUG:teuthology.orchestra.run.trial047:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:39:32.279 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T08:39:32.279 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T08:39:32.279 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T08:39:32.321 INFO:tasks.ceph.mon.a:Started 2026-02-25T08:39:32.322 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T08:39:32.322 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T08:39:32.322 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T08:39:32.325 INFO:tasks.ceph.mgr.x:Started 2026-02-25T08:39:32.325 DEBUG:tasks.ceph:set 0 configs 2026-02-25T08:39:32.325 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph config dump 2026-02-25T08:39:32.383 INFO:tasks.ceph.mon.a.trial047.stderr:2026-02-25T08:39:32.380+0000 7f643068cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.384 INFO:tasks.ceph.mon.a.trial047.stderr:2026-02-25T08:39:32.380+0000 7f643068cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.384 INFO:tasks.ceph.mon.a.trial047.stderr:2026-02-25T08:39:32.380+0000 7f643068cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.386 INFO:tasks.ceph.mgr.x.trial047.stderr:2026-02-25T08:39:32.383+0000 7ff626c27fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.400 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.394+0000 7fb75cc35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.403 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.399+0000 7fb75cc35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.435 INFO:teuthology.orchestra.run.trial047.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T08:39:32.443 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T08:39:32.444 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T08:39:32.517 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.510+0000 7fe6b1cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.519 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.516+0000 7fe6b1cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.554 INFO:teuthology.orchestra.run.trial047.stderr:adjusted tunables profile to default 2026-02-25T08:39:32.562 INFO:tasks.ceph:check_enable_crimson: True 2026-02-25T08:39:32.562 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-25T08:39:32.563 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-25T08:39:32.591 INFO:tasks.ceph.mgr.x.trial047.stderr:2026-02-25T08:39:32.588+0000 7ff626c27fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.592 INFO:tasks.ceph.mgr.x.trial047.stderr:2026-02-25T08:39:32.588+0000 7ff626c27fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.637 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.631+0000 7fc13810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.639 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:32.636+0000 7fc13810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:32.678 INFO:tasks.ceph.mon.a.trial047.stderr:2026-02-25T08:39:32.675+0000 7f6427df1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-25T08:39:32.678 INFO:tasks.ceph.mon.a.trial047.stderr:Please be aware that this feature is experimental, untested, 2026-02-25T08:39:32.678 INFO:tasks.ceph.mon.a.trial047.stderr:unsupported, and may result in data corruption, data loss, 2026-02-25T08:39:32.678 INFO:tasks.ceph.mon.a.trial047.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-25T08:39:32.678 INFO:tasks.ceph.mon.a.trial047.stderr:feature with important data. 2026-02-25T08:39:32.678 INFO:tasks.ceph.mon.a.trial047.stderr: 2026-02-25T08:39:33.441 INFO:tasks.ceph.mgr.x.trial047.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-25T08:39:33.442 INFO:tasks.ceph.mgr.x.trial047.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-25T08:39:33.442 INFO:tasks.ceph.mgr.x.trial047.stderr: from numpy import show_config as show_numpy_config 2026-02-25T08:39:33.564 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T08:39:33.565 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:33.565 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T08:39:33.630 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:33.630 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T08:39:33.695 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:39:33.695 DEBUG:teuthology.orchestra.run.trial047:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T08:39:33.758 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph osd new 4a88c0ef-1d99-4045-9137-dce175291f95 0 2026-02-25T08:39:33.873 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:33.867+0000 7f3d3915c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:33.876 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:33.872+0000 7f3d3915c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:33.917 INFO:teuthology.orchestra.run.trial047.stdout:0 2026-02-25T08:39:33.927 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph osd new 268df963-9285-40d0-8edb-e48c58f5dcbb 1 2026-02-25T08:39:34.000 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:33.993+0000 7fab8ed0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.002 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:33.999+0000 7fab8ed0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.040 INFO:teuthology.orchestra.run.trial047.stdout:1 2026-02-25T08:39:34.048 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph osd new 7f0b7530-efb8-40d7-938e-e70d51050c3a 2 2026-02-25T08:39:34.126 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:34.119+0000 7f06ee2b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.129 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:34.125+0000 7f06ee2b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.166 INFO:teuthology.orchestra.run.trial047.stdout:2 2026-02-25T08:39:34.173 DEBUG:teuthology.orchestra.run.trial047:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-25T08:39:34.200 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T08:39:34.200 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T08:39:34.243 INFO:tasks.ceph.osd.0:Started 2026-02-25T08:39:34.243 DEBUG:teuthology.orchestra.run.trial047:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-25T08:39:34.271 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T08:39:34.271 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T08:39:34.314 INFO:tasks.ceph.osd.1:Started 2026-02-25T08:39:34.315 DEBUG:teuthology.orchestra.run.trial047:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-25T08:39:34.343 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T08:39:34.343 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T08:39:34.385 INFO:tasks.ceph.osd.2:Started 2026-02-25T08:39:34.386 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:39:34.463 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:34.457+0000 7f1545da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.465 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:34.462+0000 7f1545da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.499 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:34.499 INFO:teuthology.orchestra.run.trial047.stdout:{"epoch":6,"fsid":"3ac28462-018f-46f5-99a5-a4e0dc86382c","created":"2026-02-25T08:39:32.393981+0000","modified":"2026-02-25T08:39:34.163595+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:39:34.163595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4a88c0ef-1d99-4045-9137-dce175291f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"268df963-9285-40d0-8edb-e48c58f5dcbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f0b7530-efb8-40d7-938e-e70d51050c3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:39:34.509 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T08:39:34.509 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T08:39:34.811 DEBUG:teuthology.orchestra.run.trial047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:39:34.879 INFO:teuthology.misc.health.trial047.stderr:2026-02-25T08:39:34.873+0000 7fc9180cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.881 INFO:teuthology.misc.health.trial047.stderr:2026-02-25T08:39:34.878+0000 7fc9180cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:34.915 INFO:teuthology.misc.health.trial047.stdout: 2026-02-25T08:39:34.915 INFO:teuthology.misc.health.trial047.stdout:{"epoch":6,"fsid":"3ac28462-018f-46f5-99a5-a4e0dc86382c","created":"2026-02-25T08:39:32.393981+0000","modified":"2026-02-25T08:39:34.163595+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:39:34.163595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4a88c0ef-1d99-4045-9137-dce175291f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"268df963-9285-40d0-8edb-e48c58f5dcbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f0b7530-efb8-40d7-938e-e70d51050c3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:39:34.922 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-25T08:39:35.871 INFO:tasks.ceph.mgr.x.trial047.stderr:2026-02-25T08:39:35.868+0000 7ff54206c640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-25T08:39:41.225 DEBUG:teuthology.orchestra.run.trial047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:39:41.297 INFO:teuthology.misc.health.trial047.stderr:2026-02-25T08:39:41.291+0000 7fa39a374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:41.300 INFO:teuthology.misc.health.trial047.stderr:2026-02-25T08:39:41.297+0000 7fa39a374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:41.424 INFO:teuthology.misc.health.trial047.stdout: 2026-02-25T08:39:41.424 INFO:teuthology.misc.health.trial047.stdout:{"epoch":8,"fsid":"3ac28462-018f-46f5-99a5-a4e0dc86382c","created":"2026-02-25T08:39:32.393981+0000","modified":"2026-02-25T08:39:40.736048+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:39:34.163595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"4a88c0ef-1d99-4045-9137-dce175291f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"268df963-9285-40d0-8edb-e48c58f5dcbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f0b7530-efb8-40d7-938e-e70d51050c3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:39:41.436 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-25T08:39:47.737 DEBUG:teuthology.orchestra.run.trial047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:39:47.809 INFO:teuthology.misc.health.trial047.stderr:2026-02-25T08:39:47.805+0000 7f8b6b8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:47.812 INFO:teuthology.misc.health.trial047.stderr:2026-02-25T08:39:47.809+0000 7f8b6b8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:47.924 INFO:teuthology.misc.health.trial047.stdout: 2026-02-25T08:39:47.924 INFO:teuthology.misc.health.trial047.stdout:{"epoch":13,"fsid":"3ac28462-018f-46f5-99a5-a4e0dc86382c","created":"2026-02-25T08:39:32.393981+0000","modified":"2026-02-25T08:39:45.757274+0000","last_up_change":"2026-02-25T08:39:43.746343+0000","last_in_change":"2026-02-25T08:39:34.163595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:39:42.881297+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a88c0ef-1d99-4045-9137-dce175291f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6808","nonce":1498602973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6809","nonce":1498602973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6813","nonce":1498602973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6814","nonce":1498602973}]},"public_addr":"10.20.193.47:6808/1498602973","cluster_addr":"10.20.193.47:6809/1498602973","heartbeat_back_addr":"10.20.193.47:6813/1498602973","heartbeat_front_addr":"10.20.193.47:6814/1498602973","state":["exists","up"]},{"osd":1,"uuid":"268df963-9285-40d0-8edb-e48c58f5dcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6805","nonce":4005732361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6806","nonce":4005732361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6811","nonce":4005732361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6810","nonce":4005732361}]},"public_addr":"10.20.193.47:6805/4005732361","cluster_addr":"10.20.193.47:6806/4005732361","heartbeat_back_addr":"10.20.193.47:6811/4005732361","heartbeat_front_addr":"10.20.193.47:6810/4005732361","state":["exists","up"]},{"osd":2,"uuid":"7f0b7530-efb8-40d7-938e-e70d51050c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6802","nonce":654768586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6803","nonce":654768586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6807","nonce":654768586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6804","nonce":654768586}]},"public_addr":"10.20.193.47:6802/654768586","cluster_addr":"10.20.193.47:6803/654768586","heartbeat_back_addr":"10.20.193.47:6807/654768586","heartbeat_front_addr":"10.20.193.47:6804/654768586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:39:47.932 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-25T08:39:47.932 INFO:tasks.ceph:Creating RBD pool 2026-02-25T08:39:47.932 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T08:39:48.007 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:48.000+0000 7f745e05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:48.009 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:48.006+0000 7f745e05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:48.771 INFO:teuthology.orchestra.run.trial047.stderr:pool 'rbd' created 2026-02-25T08:39:48.785 DEBUG:teuthology.orchestra.run.trial047:> rbd --cluster ceph pool init rbd 2026-02-25T08:39:48.830 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:48.828+0000 7f751defb240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:48.830 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:48.828+0000 7f751defb240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:48.834 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:48.831+0000 7f751defb240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:51.811 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T08:39:51.812 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T08:39:51.813 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T08:39:51.891 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:51.885+0000 7f6262e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:51.893 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:51.891+0000 7f6262e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.003 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:52.012 INFO:teuthology.orchestra.run.trial047.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-25T08:39:52.013 INFO:tasks.ceph_manager:config epoch is 1 2026-02-25T08:39:52.013 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T08:39:52.013 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T08:39:52.013 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T08:39:52.088 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.083+0000 7f6129ac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.090 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.088+0000 7f6129ac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.236 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:52.244 INFO:teuthology.orchestra.run.trial047.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6800","nonce":3470612224},{"type":"v1","addr":"10.20.193.47:6801","nonce":3470612224}]},"active_addr":"10.20.193.47:6801/3470612224","active_change":"2026-02-25T08:39:34.860327+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.47:0","nonce":295572939}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.47:0","nonce":4294653344}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.47:0","nonce":2172778442}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.47:0","nonce":1738007962}]}]} 2026-02-25T08:39:52.248 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T08:39:52.248 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:39:52.248 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:39:52.325 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.320+0000 7fa99f993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.328 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.325+0000 7fa99f993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.442 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:52.442 INFO:teuthology.orchestra.run.trial047.stdout:{"epoch":17,"fsid":"3ac28462-018f-46f5-99a5-a4e0dc86382c","created":"2026-02-25T08:39:32.393981+0000","modified":"2026-02-25T08:39:51.777855+0000","last_up_change":"2026-02-25T08:39:43.746343+0000","last_in_change":"2026-02-25T08:39:34.163595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:39:42.881297+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-25T08:39:48.121029+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a88c0ef-1d99-4045-9137-dce175291f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6808","nonce":1498602973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6809","nonce":1498602973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6813","nonce":1498602973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6814","nonce":1498602973}]},"public_addr":"10.20.193.47:6808/1498602973","cluster_addr":"10.20.193.47:6809/1498602973","heartbeat_back_addr":"10.20.193.47:6813/1498602973","heartbeat_front_addr":"10.20.193.47:6814/1498602973","state":["exists","up"]},{"osd":1,"uuid":"268df963-9285-40d0-8edb-e48c58f5dcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6805","nonce":4005732361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6806","nonce":4005732361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6811","nonce":4005732361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6810","nonce":4005732361}]},"public_addr":"10.20.193.47:6805/4005732361","cluster_addr":"10.20.193.47:6806/4005732361","heartbeat_back_addr":"10.20.193.47:6811/4005732361","heartbeat_front_addr":"10.20.193.47:6810/4005732361","state":["exists","up"]},{"osd":2,"uuid":"7f0b7530-efb8-40d7-938e-e70d51050c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6802","nonce":654768586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6803","nonce":654768586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6807","nonce":654768586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6804","nonce":654768586}]},"public_addr":"10.20.193.47:6802/654768586","cluster_addr":"10.20.193.47:6803/654768586","heartbeat_back_addr":"10.20.193.47:6807/654768586","heartbeat_front_addr":"10.20.193.47:6804/654768586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:39:52.453 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T08:39:52.453 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:39:52.529 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.524+0000 7f32f6851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.531 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.529+0000 7f32f6851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.644 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:52.644 INFO:teuthology.orchestra.run.trial047.stdout:{"epoch":17,"fsid":"3ac28462-018f-46f5-99a5-a4e0dc86382c","created":"2026-02-25T08:39:32.393981+0000","modified":"2026-02-25T08:39:51.777855+0000","last_up_change":"2026-02-25T08:39:43.746343+0000","last_in_change":"2026-02-25T08:39:34.163595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:39:42.881297+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-25T08:39:48.121029+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a88c0ef-1d99-4045-9137-dce175291f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6808","nonce":1498602973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6809","nonce":1498602973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6813","nonce":1498602973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6814","nonce":1498602973}]},"public_addr":"10.20.193.47:6808/1498602973","cluster_addr":"10.20.193.47:6809/1498602973","heartbeat_back_addr":"10.20.193.47:6813/1498602973","heartbeat_front_addr":"10.20.193.47:6814/1498602973","state":["exists","up"]},{"osd":1,"uuid":"268df963-9285-40d0-8edb-e48c58f5dcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6805","nonce":4005732361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6806","nonce":4005732361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6811","nonce":4005732361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6810","nonce":4005732361}]},"public_addr":"10.20.193.47:6805/4005732361","cluster_addr":"10.20.193.47:6806/4005732361","heartbeat_back_addr":"10.20.193.47:6811/4005732361","heartbeat_front_addr":"10.20.193.47:6810/4005732361","state":["exists","up"]},{"osd":2,"uuid":"7f0b7530-efb8-40d7-938e-e70d51050c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6802","nonce":654768586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6803","nonce":654768586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6807","nonce":654768586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6804","nonce":654768586}]},"public_addr":"10.20.193.47:6802/654768586","cluster_addr":"10.20.193.47:6803/654768586","heartbeat_back_addr":"10.20.193.47:6807/654768586","heartbeat_front_addr":"10.20.193.47:6804/654768586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:39:52.655 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T08:39:52.656 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T08:39:52.656 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T08:39:52.738 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.733+0000 7fc65d5cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.739 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.733+0000 7f66b2d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.739 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.733+0000 7fee1cbdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.741 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.738+0000 7fee1cbdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.741 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.738+0000 7f66b2d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.741 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.738+0000 7fc65d5cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.759 INFO:teuthology.orchestra.run.trial047.stdout:42949672969 2026-02-25T08:39:52.759 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:39:52.760 INFO:teuthology.orchestra.run.trial047.stdout:38654705674 2026-02-25T08:39:52.761 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:39:52.762 INFO:teuthology.orchestra.run.trial047.stdout:47244640264 2026-02-25T08:39:52.762 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:39:52.881 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.876+0000 7f2f3eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.882 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.876+0000 7f799110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.882 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.876+0000 7f0236d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.884 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.881+0000 7f799110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.884 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.881+0000 7f2f3eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.884 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:52.882+0000 7f0236d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:52.996 INFO:teuthology.orchestra.run.trial047.stdout:38654705673 2026-02-25T08:39:53.000 INFO:teuthology.orchestra.run.trial047.stdout:42949672964 2026-02-25T08:39:53.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.2 2026-02-25T08:39:53.008 INFO:teuthology.orchestra.run.trial047.stdout:47244640264 2026-02-25T08:39:53.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.1 2026-02-25T08:39:53.019 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640264 for osd.0 2026-02-25T08:39:53.019 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:54.008 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:39:54.012 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:39:54.085 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.079+0000 7f420870b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.087 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.085+0000 7f420870b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.128 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.123+0000 7f69aba5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.130 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.128+0000 7f69aba5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.198 INFO:teuthology.orchestra.run.trial047.stdout:38654705674 2026-02-25T08:39:54.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.2 2026-02-25T08:39:54.209 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:54.241 INFO:teuthology.orchestra.run.trial047.stdout:42949672969 2026-02-25T08:39:54.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.1 2026-02-25T08:39:54.251 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:54.252 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:39:54.252 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:39:54.366 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.361+0000 7eff9bf0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.368 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.366+0000 7eff9bf0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.478 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:54.478 INFO:teuthology.orchestra.run.trial047.stderr:dumped all 2026-02-25T08:39:54.486 INFO:teuthology.orchestra.run.trial047.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-25T08:39:52.868110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2812,"kb_used_data":2812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497156,"statfs":{"total":160255967232,"available":160253087744,"internally_reserved":0,"allocated":2879488,"data_stored":2879488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.099278"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.789656+0000","last_change":"2026-02-25T08:39:51.793690+0000","last_active":"2026-02-25T08:39:51.789656+0000","last_peered":"2026-02-25T08:39:51.789656+0000","last_clean":"2026-02-25T08:39:51.789656+0000","last_became_active":"2026-02-25T08:39:49.814925+0000","last_became_peered":"2026-02-25T08:39:49.814925+0000","last_unstale":"2026-02-25T08:39:51.789656+0000","last_undegraded":"2026-02-25T08:39:51.789656+0000","last_fullsized":"2026-02-25T08:39:51.789656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.791827+0000","last_change":"2026-02-25T08:39:51.794622+0000","last_active":"2026-02-25T08:39:51.791827+0000","last_peered":"2026-02-25T08:39:51.791827+0000","last_clean":"2026-02-25T08:39:51.791827+0000","last_became_active":"2026-02-25T08:39:49.808924+0000","last_became_peered":"2026-02-25T08:39:49.808924+0000","last_unstale":"2026-02-25T08:39:51.791827+0000","last_undegraded":"2026-02-25T08:39:51.791827+0000","last_fullsized":"2026-02-25T08:39:51.791827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.793260+0000","last_change":"2026-02-25T08:39:51.797616+0000","last_active":"2026-02-25T08:39:51.793260+0000","last_peered":"2026-02-25T08:39:51.793260+0000","last_clean":"2026-02-25T08:39:51.793260+0000","last_became_active":"2026-02-25T08:39:49.807974+0000","last_became_peered":"2026-02-25T08:39:49.807974+0000","last_unstale":"2026-02-25T08:39:51.793260+0000","last_undegraded":"2026-02-25T08:39:51.793260+0000","last_fullsized":"2026-02-25T08:39:51.793260+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.792853+0000","last_change":"2026-02-25T08:39:51.797568+0000","last_active":"2026-02-25T08:39:51.792853+0000","last_peered":"2026-02-25T08:39:51.792853+0000","last_clean":"2026-02-25T08:39:51.792853+0000","last_became_active":"2026-02-25T08:39:49.817503+0000","last_became_peered":"2026-02-25T08:39:49.817503+0000","last_unstale":"2026-02-25T08:39:51.792853+0000","last_undegraded":"2026-02-25T08:39:51.792853+0000","last_fullsized":"2026-02-25T08:39:51.792853+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.796471+0000","last_change":"2026-02-25T08:39:51.796471+0000","last_active":"2026-02-25T08:39:51.796471+0000","last_peered":"2026-02-25T08:39:51.796471+0000","last_clean":"2026-02-25T08:39:51.796471+0000","last_became_active":"2026-02-25T08:39:49.827668+0000","last_became_peered":"2026-02-25T08:39:49.827668+0000","last_unstale":"2026-02-25T08:39:51.796471+0000","last_undegraded":"2026-02-25T08:39:51.796471+0000","last_fullsized":"2026-02-25T08:39:51.796471+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.792507+0000","last_change":"2026-02-25T08:39:51.794859+0000","last_active":"2026-02-25T08:39:51.792507+0000","last_peered":"2026-02-25T08:39:51.792507+0000","last_clean":"2026-02-25T08:39:51.792507+0000","last_became_active":"2026-02-25T08:39:49.826833+0000","last_became_peered":"2026-02-25T08:39:49.826833+0000","last_unstale":"2026-02-25T08:39:51.792507+0000","last_undegraded":"2026-02-25T08:39:51.792507+0000","last_fullsized":"2026-02-25T08:39:51.792507+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.789727+0000","last_change":"2026-02-25T08:39:51.792927+0000","last_active":"2026-02-25T08:39:51.789727+0000","last_peered":"2026-02-25T08:39:51.789727+0000","last_clean":"2026-02-25T08:39:51.789727+0000","last_became_active":"2026-02-25T08:39:49.831241+0000","last_became_peered":"2026-02-25T08:39:49.831241+0000","last_unstale":"2026-02-25T08:39:51.789727+0000","last_undegraded":"2026-02-25T08:39:51.789727+0000","last_fullsized":"2026-02-25T08:39:51.789727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.791836+0000","last_change":"2026-02-25T08:39:51.794944+0000","last_active":"2026-02-25T08:39:51.791836+0000","last_peered":"2026-02-25T08:39:51.791836+0000","last_clean":"2026-02-25T08:39:51.791836+0000","last_became_active":"2026-02-25T08:39:49.810544+0000","last_became_peered":"2026-02-25T08:39:49.810544+0000","last_unstale":"2026-02-25T08:39:51.791836+0000","last_undegraded":"2026-02-25T08:39:51.791836+0000","last_fullsized":"2026-02-25T08:39:51.791836+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.791256+0000","last_change":"2026-02-25T08:39:44.791638+0000","last_active":"2026-02-25T08:39:51.791256+0000","last_peered":"2026-02-25T08:39:51.791256+0000","last_clean":"2026-02-25T08:39:51.791256+0000","last_became_active":"2026-02-25T08:39:44.782576+0000","last_became_peered":"2026-02-25T08:39:44.782576+0000","last_unstale":"2026-02-25T08:39:51.791256+0000","last_undegraded":"2026-02-25T08:39:51.791256+0000","last_fullsized":"2026-02-25T08:39:51.791256+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_clean_scrub_stamp":"2026-02-25T08:39:43.746343+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253063168,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417340928,"internally_reserved":0,"allocated":1314816,"data_stored":1314816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1180,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165476,"statfs":{"total":53418655744,"available":53417447424,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":356,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166300,"statfs":{"total":53418655744,"available":53418291200,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417340928,"internally_reserved":0,"allocated":1314816,"data_stored":1314816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417439232,"internally_reserved":0,"allocated":1216512,"data_stored":1216512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417340928,"internally_reserved":0,"allocated":1314816,"data_stored":1314816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418283008,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:39:54.488 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:39:54.562 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.558+0000 7fd38d0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.565 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.562+0000 7fd38d0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.674 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:54.674 INFO:teuthology.orchestra.run.trial047.stderr:dumped all 2026-02-25T08:39:54.683 INFO:teuthology.orchestra.run.trial047.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-25T08:39:52.868110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2812,"kb_used_data":2812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497156,"statfs":{"total":160255967232,"available":160253087744,"internally_reserved":0,"allocated":2879488,"data_stored":2879488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.099278"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.789656+0000","last_change":"2026-02-25T08:39:51.793690+0000","last_active":"2026-02-25T08:39:51.789656+0000","last_peered":"2026-02-25T08:39:51.789656+0000","last_clean":"2026-02-25T08:39:51.789656+0000","last_became_active":"2026-02-25T08:39:49.814925+0000","last_became_peered":"2026-02-25T08:39:49.814925+0000","last_unstale":"2026-02-25T08:39:51.789656+0000","last_undegraded":"2026-02-25T08:39:51.789656+0000","last_fullsized":"2026-02-25T08:39:51.789656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.791827+0000","last_change":"2026-02-25T08:39:51.794622+0000","last_active":"2026-02-25T08:39:51.791827+0000","last_peered":"2026-02-25T08:39:51.791827+0000","last_clean":"2026-02-25T08:39:51.791827+0000","last_became_active":"2026-02-25T08:39:49.808924+0000","last_became_peered":"2026-02-25T08:39:49.808924+0000","last_unstale":"2026-02-25T08:39:51.791827+0000","last_undegraded":"2026-02-25T08:39:51.791827+0000","last_fullsized":"2026-02-25T08:39:51.791827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.793260+0000","last_change":"2026-02-25T08:39:51.797616+0000","last_active":"2026-02-25T08:39:51.793260+0000","last_peered":"2026-02-25T08:39:51.793260+0000","last_clean":"2026-02-25T08:39:51.793260+0000","last_became_active":"2026-02-25T08:39:49.807974+0000","last_became_peered":"2026-02-25T08:39:49.807974+0000","last_unstale":"2026-02-25T08:39:51.793260+0000","last_undegraded":"2026-02-25T08:39:51.793260+0000","last_fullsized":"2026-02-25T08:39:51.793260+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.792853+0000","last_change":"2026-02-25T08:39:51.797568+0000","last_active":"2026-02-25T08:39:51.792853+0000","last_peered":"2026-02-25T08:39:51.792853+0000","last_clean":"2026-02-25T08:39:51.792853+0000","last_became_active":"2026-02-25T08:39:49.817503+0000","last_became_peered":"2026-02-25T08:39:49.817503+0000","last_unstale":"2026-02-25T08:39:51.792853+0000","last_undegraded":"2026-02-25T08:39:51.792853+0000","last_fullsized":"2026-02-25T08:39:51.792853+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.796471+0000","last_change":"2026-02-25T08:39:51.796471+0000","last_active":"2026-02-25T08:39:51.796471+0000","last_peered":"2026-02-25T08:39:51.796471+0000","last_clean":"2026-02-25T08:39:51.796471+0000","last_became_active":"2026-02-25T08:39:49.827668+0000","last_became_peered":"2026-02-25T08:39:49.827668+0000","last_unstale":"2026-02-25T08:39:51.796471+0000","last_undegraded":"2026-02-25T08:39:51.796471+0000","last_fullsized":"2026-02-25T08:39:51.796471+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.792507+0000","last_change":"2026-02-25T08:39:51.794859+0000","last_active":"2026-02-25T08:39:51.792507+0000","last_peered":"2026-02-25T08:39:51.792507+0000","last_clean":"2026-02-25T08:39:51.792507+0000","last_became_active":"2026-02-25T08:39:49.826833+0000","last_became_peered":"2026-02-25T08:39:49.826833+0000","last_unstale":"2026-02-25T08:39:51.792507+0000","last_undegraded":"2026-02-25T08:39:51.792507+0000","last_fullsized":"2026-02-25T08:39:51.792507+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.789727+0000","last_change":"2026-02-25T08:39:51.792927+0000","last_active":"2026-02-25T08:39:51.789727+0000","last_peered":"2026-02-25T08:39:51.789727+0000","last_clean":"2026-02-25T08:39:51.789727+0000","last_became_active":"2026-02-25T08:39:49.831241+0000","last_became_peered":"2026-02-25T08:39:49.831241+0000","last_unstale":"2026-02-25T08:39:51.789727+0000","last_undegraded":"2026-02-25T08:39:51.789727+0000","last_fullsized":"2026-02-25T08:39:51.789727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.791836+0000","last_change":"2026-02-25T08:39:51.794944+0000","last_active":"2026-02-25T08:39:51.791836+0000","last_peered":"2026-02-25T08:39:51.791836+0000","last_clean":"2026-02-25T08:39:51.791836+0000","last_became_active":"2026-02-25T08:39:49.810544+0000","last_became_peered":"2026-02-25T08:39:49.810544+0000","last_unstale":"2026-02-25T08:39:51.791836+0000","last_undegraded":"2026-02-25T08:39:51.791836+0000","last_fullsized":"2026-02-25T08:39:51.791836+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:39:51.791256+0000","last_change":"2026-02-25T08:39:44.791638+0000","last_active":"2026-02-25T08:39:51.791256+0000","last_peered":"2026-02-25T08:39:51.791256+0000","last_clean":"2026-02-25T08:39:51.791256+0000","last_became_active":"2026-02-25T08:39:44.782576+0000","last_became_peered":"2026-02-25T08:39:44.782576+0000","last_unstale":"2026-02-25T08:39:51.791256+0000","last_undegraded":"2026-02-25T08:39:51.791256+0000","last_fullsized":"2026-02-25T08:39:51.791256+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_clean_scrub_stamp":"2026-02-25T08:39:43.746343+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253063168,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417340928,"internally_reserved":0,"allocated":1314816,"data_stored":1314816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1180,"kb_used_data":1180,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165476,"statfs":{"total":53418655744,"available":53417447424,"internally_reserved":0,"allocated":1208320,"data_stored":1208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":356,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166300,"statfs":{"total":53418655744,"available":53418291200,"internally_reserved":0,"allocated":364544,"data_stored":364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417340928,"internally_reserved":0,"allocated":1314816,"data_stored":1314816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417439232,"internally_reserved":0,"allocated":1216512,"data_stored":1216512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417340928,"internally_reserved":0,"allocated":1314816,"data_stored":1314816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418283008,"internally_reserved":0,"allocated":372736,"data_stored":372736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:39:54.685 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:39:54.685 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T08:39:54.685 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T08:39:54.685 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T08:39:54.759 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.755+0000 7ff97a61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.762 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:39:54.759+0000 7ff97a61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:39:54.899 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:39:54.899 INFO:teuthology.orchestra.run.trial047.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T08:39:54.909 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T08:39:54.909 INFO:teuthology.run_tasks:Running task workunit... 2026-02-25T08:39:54.928 INFO:tasks.workunit:Pulling workunits from ref c32469671c6b0222f786dd36c34eb702f780a11d 2026-02-25T08:39:54.928 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-25T08:39:54.928 DEBUG:teuthology.orchestra.run.trial047:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-25T08:39:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:39:54.944 INFO:teuthology.orchestra.run.trial047.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-25T08:39:54.945 DEBUG:teuthology.orchestra.run.trial047:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T08:39:55.001 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-25T08:39:55.001 DEBUG:teuthology.orchestra.run.trial047:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-25T08:39:55.057 INFO:tasks.workunit:timeout=3h 2026-02-25T08:39:55.057 INFO:tasks.workunit:cleanup=True 2026-02-25T08:39:55.058 DEBUG:teuthology.orchestra.run.trial047:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c32469671c6b0222f786dd36c34eb702f780a11d 2026-02-25T08:39:55.121 INFO:tasks.workunit.client.0.trial047.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr:Note: switching to 'c32469671c6b0222f786dd36c34eb702f780a11d'. 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr: 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr:state without impacting any branches by switching back to a branch. 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr: 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-25T08:40:18.830 INFO:tasks.workunit.client.0.trial047.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr: 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr: git switch -c 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr: 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr:Or undo this operation with: 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr: 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr: git switch - 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr: 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr: 2026-02-25T08:40:18.831 INFO:tasks.workunit.client.0.trial047.stderr:HEAD is now at c32469671c6 Merge pull request #66090 from rhcs-dashboard/73674-create-rule-certmgr 2026-02-25T08:40:18.831 DEBUG:teuthology.orchestra.run.trial047:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-25T08:40:18.888 INFO:tasks.workunit.client.0.trial047.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-25T08:40:18.890 INFO:tasks.workunit.client.0.trial047.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-25T08:40:18.890 INFO:tasks.workunit.client.0.trial047.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-25T08:40:18.979 INFO:tasks.workunit.client.0.trial047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-25T08:40:19.011 INFO:tasks.workunit.client.0.trial047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-25T08:40:19.039 INFO:tasks.workunit.client.0.trial047.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-25T08:40:19.040 INFO:tasks.workunit.client.0.trial047.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-25T08:40:19.040 INFO:tasks.workunit.client.0.trial047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-25T08:40:19.066 INFO:tasks.workunit.client.0.trial047.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-25T08:40:19.070 DEBUG:teuthology.orchestra.run.trial047:> set -ex 2026-02-25T08:40:19.070 DEBUG:teuthology.orchestra.run.trial047:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-25T08:40:19.127 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2026-02-25T08:40:19.130 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2026-02-25T08:40:19.130 DEBUG:teuthology.orchestra.run.trial047:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c32469671c6b0222f786dd36c34eb702f780a11d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2026-02-25T08:40:19.201 INFO:tasks.workunit.client.0.trial047.stdout:Running main() from gmock_main.cc 2026-02-25T08:40:19.201 INFO:tasks.workunit.client.0.trial047.stdout:[==========] Running 71 tests from 1 test suite. 2026-02-25T08:40:19.201 INFO:tasks.workunit.client.0.trial047.stdout:[----------] Global test environment set-up. 2026-02-25T08:40:19.201 INFO:tasks.workunit.client.0.trial047.stdout:[----------] 71 tests from TestClsRbd 2026-02-25T08:40:20.946 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_all_features 2026-02-25T08:40:21.129 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_all_features (183 ms) 2026-02-25T08:40:21.130 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.copyup 2026-02-25T08:40:21.253 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.copyup (123 ms) 2026-02-25T08:40:21.253 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.sparse_copyup 2026-02-25T08:40:21.348 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.sparse_copyup (95 ms) 2026-02-25T08:40:21.348 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_and_set_id 2026-02-25T08:40:21.400 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_and_set_id (52 ms) 2026-02-25T08:40:21.400 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.add_remove_child 2026-02-25T08:40:21.492 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.add_remove_child (91 ms) 2026-02-25T08:40:21.492 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.directory_methods 2026-02-25T08:40:21.745 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.directory_methods (253 ms) 2026-02-25T08:40:21.746 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.create 2026-02-25T08:40:21.863 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.create (117 ms) 2026-02-25T08:40:21.863 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_features 2026-02-25T08:40:21.878 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_features (15 ms) 2026-02-25T08:40:21.878 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_object_prefix 2026-02-25T08:40:21.893 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_object_prefix (14 ms) 2026-02-25T08:40:21.893 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2026-02-25T08:40:21.912 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_create_timestamp (18 ms) 2026-02-25T08:40:21.912 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2026-02-25T08:40:21.927 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_access_timestamp (14 ms) 2026-02-25T08:40:21.927 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2026-02-25T08:40:21.941 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_modify_timestamp (14 ms) 2026-02-25T08:40:21.941 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_data_pool 2026-02-25T08:40:21.985 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_data_pool (43 ms) 2026-02-25T08:40:21.985 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.get_size 2026-02-25T08:40:22.025 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.get_size (39 ms) 2026-02-25T08:40:22.025 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.set_size 2026-02-25T08:40:22.070 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.set_size (45 ms) 2026-02-25T08:40:22.070 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.protection_status 2026-02-25T08:40:22.243 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.protection_status (173 ms) 2026-02-25T08:40:22.244 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.snapshot_limits 2026-02-25T08:40:22.326 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.snapshot_limits (82 ms) 2026-02-25T08:40:22.326 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.parents_v1 2026-02-25T08:40:22.789 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.parents_v1 (463 ms) 2026-02-25T08:40:22.789 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.parents_v2 2026-02-25T08:40:23.097 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.parents_v2 (307 ms) 2026-02-25T08:40:23.097 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.snapshots 2026-02-25T08:40:23.333 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.snapshots (236 ms) 2026-02-25T08:40:23.333 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.snapid_race 2026-02-25T08:40:23.372 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.snapid_race (38 ms) 2026-02-25T08:40:23.372 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.stripingv2 2026-02-25T08:40:23.455 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.stripingv2 (83 ms) 2026-02-25T08:40:23.455 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.object_map_save 2026-02-25T08:40:23.469 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.object_map_save (13 ms) 2026-02-25T08:40:23.469 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.object_map_resize 2026-02-25T08:40:23.514 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.object_map_resize (45 ms) 2026-02-25T08:40:23.514 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.object_map_update 2026-02-25T08:40:23.557 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.object_map_update (42 ms) 2026-02-25T08:40:23.557 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2026-02-25T08:40:23.559 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.object_map_load_enoent (2 ms) 2026-02-25T08:40:23.559 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2026-02-25T08:40:23.600 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.object_map_snap_add (40 ms) 2026-02-25T08:40:23.600 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2026-02-25T08:40:23.638 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.object_map_snap_remove (38 ms) 2026-02-25T08:40:23.638 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.flags 2026-02-25T08:40:23.687 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.flags (49 ms) 2026-02-25T08:40:23.687 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.metadata 2026-02-25T08:40:24.035 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.metadata (347 ms) 2026-02-25T08:40:24.035 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.set_features 2026-02-25T08:40:24.097 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.set_features (61 ms) 2026-02-25T08:40:24.097 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.mirror 2026-02-25T08:40:24.429 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.mirror (332 ms) 2026-02-25T08:40:24.429 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.mirror_image 2026-02-25T08:40:24.579 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.mirror_image (149 ms) 2026-02-25T08:40:24.579 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.mirror_image_status 2026-02-25T08:40:54.563 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.mirror_image_status (29983 ms) 2026-02-25T08:40:54.563 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.mirror_image_map 2026-02-25T08:40:55.553 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.mirror_image_map (990 ms) 2026-02-25T08:40:55.554 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.mirror_instances 2026-02-25T08:40:55.622 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.mirror_instances (68 ms) 2026-02-25T08:40:55.622 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2026-02-25T08:40:55.744 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.mirror_snapshot (122 ms) 2026-02-25T08:40:55.744 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_dir_list 2026-02-25T08:40:55.768 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_dir_list (23 ms) 2026-02-25T08:40:55.768 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_dir_add 2026-02-25T08:40:55.785 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_dir_add (17 ms) 2026-02-25T08:40:55.786 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2026-02-25T08:40:55.812 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.dir_add_already_existing (26 ms) 2026-02-25T08:40:55.812 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_dir_rename 2026-02-25T08:40:55.871 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_dir_rename (58 ms) 2026-02-25T08:40:55.872 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_dir_remove 2026-02-25T08:40:55.902 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_dir_remove (31 ms) 2026-02-25T08:40:55.902 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2026-02-25T08:40:55.939 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (36 ms) 2026-02-25T08:40:55.939 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_image_add 2026-02-25T08:40:55.958 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_image_add (18 ms) 2026-02-25T08:40:55.958 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_image_remove 2026-02-25T08:40:55.988 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_image_remove (29 ms) 2026-02-25T08:40:55.988 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_image_list 2026-02-25T08:40:56.013 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_image_list (24 ms) 2026-02-25T08:40:56.013 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_image_clean 2026-02-25T08:40:56.055 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_image_clean (42 ms) 2026-02-25T08:40:56.055 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.image_group_add 2026-02-25T08:40:56.081 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.image_group_add (25 ms) 2026-02-25T08:40:56.081 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.image_group_remove 2026-02-25T08:40:56.118 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.image_group_remove (37 ms) 2026-02-25T08:40:56.118 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.image_group_get 2026-02-25T08:40:56.143 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.image_group_get (24 ms) 2026-02-25T08:40:56.143 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2026-02-25T08:40:56.157 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (14 ms) 2026-02-25T08:40:56.158 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2026-02-25T08:40:56.171 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (13 ms) 2026-02-25T08:40:56.171 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2026-02-25T08:40:56.198 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (27 ms) 2026-02-25T08:40:56.199 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2026-02-25T08:40:56.225 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (26 ms) 2026-02-25T08:40:56.225 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_set 2026-02-25T08:40:56.246 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_set (20 ms) 2026-02-25T08:40:56.246 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_list 2026-02-25T08:40:56.317 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_list (71 ms) 2026-02-25T08:40:56.317 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2026-02-25T08:40:56.506 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (189 ms) 2026-02-25T08:40:56.506 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2026-02-25T08:40:58.446 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (1940 ms) 2026-02-25T08:40:58.447 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_remove 2026-02-25T08:40:58.475 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_remove (28 ms) 2026-02-25T08:40:58.475 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_remove_without_order 2026-02-25T08:40:58.514 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_remove_without_order (39 ms) 2026-02-25T08:40:58.514 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2026-02-25T08:40:58.534 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (19 ms) 2026-02-25T08:40:58.534 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.trash_methods 2026-02-25T08:40:58.598 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.trash_methods (63 ms) 2026-02-25T08:40:58.598 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.op_features 2026-02-25T08:40:58.661 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.op_features (63 ms) 2026-02-25T08:40:58.662 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.clone_parent 2026-02-25T08:40:58.887 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.clone_parent (225 ms) 2026-02-25T08:40:58.887 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2026-02-25T08:40:58.991 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.clone_parent_ns (104 ms) 2026-02-25T08:40:58.991 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.clone_child 2026-02-25T08:40:59.115 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.clone_child (124 ms) 2026-02-25T08:40:59.116 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.namespace_methods 2026-02-25T08:40:59.174 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.namespace_methods (58 ms) 2026-02-25T08:40:59.174 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.migration 2026-02-25T08:40:59.268 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.migration (94 ms) 2026-02-25T08:40:59.268 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.migration_v1 2026-02-25T08:40:59.355 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.migration_v1 (87 ms) 2026-02-25T08:40:59.356 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2026-02-25T08:41:00.426 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1070 ms) 2026-02-25T08:41:00.426 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsRbd.sparsify 2026-02-25T08:41:00.646 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsRbd.sparsify (220 ms) 2026-02-25T08:41:01.432 INFO:tasks.workunit.client.0.trial047.stdout:[----------] 71 tests from TestClsRbd (39700 ms total) 2026-02-25T08:41:01.432 INFO:tasks.workunit.client.0.trial047.stdout: 2026-02-25T08:41:01.432 INFO:tasks.workunit.client.0.trial047.stdout:[----------] Global test environment tear-down 2026-02-25T08:41:01.432 INFO:tasks.workunit.client.0.trial047.stdout:[==========] 71 tests from 1 test suite ran. (42230 ms total) 2026-02-25T08:41:01.432 INFO:tasks.workunit.client.0.trial047.stdout:[ PASSED ] 71 tests. 2026-02-25T08:41:01.435 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T08:41:01.435 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T08:41:01.504 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2026-02-25T08:41:01.507 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2026-02-25T08:41:01.508 DEBUG:teuthology.orchestra.run.trial047:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c32469671c6b0222f786dd36c34eb702f780a11d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2026-02-25T08:41:01.579 INFO:tasks.workunit.client.0.trial047.stdout:Running main() from gmock_main.cc 2026-02-25T08:41:01.579 INFO:tasks.workunit.client.0.trial047.stdout:[==========] Running 11 tests from 1 test suite. 2026-02-25T08:41:01.579 INFO:tasks.workunit.client.0.trial047.stdout:[----------] Global test environment set-up. 2026-02-25T08:41:01.580 INFO:tasks.workunit.client.0.trial047.stdout:[----------] 11 tests from ClsLock 2026-02-25T08:41:01.580 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestMultiLocking 2026-02-25T08:41:03.731 INFO:tasks.workunit.client.0.trial047.stdout:lock: mylock 2026-02-25T08:41:03.731 INFO:tasks.workunit.client.0.trial047.stdout: lock_type: exclusive 2026-02-25T08:41:03.731 INFO:tasks.workunit.client.0.trial047.stdout: tag: 2026-02-25T08:41:03.731 INFO:tasks.workunit.client.0.trial047.stdout: lockers: 2026-02-25T08:41:03.731 INFO:tasks.workunit.client.0.trial047.stdout: client.4204 expiration=2026-02-25T08:43:03.690873+0000 addr=v1:10.20.193.47:0/3358350364 cookie= 2026-02-25T08:41:03.763 INFO:tasks.workunit.client.0.trial047.stdout:lock: mylock 2026-02-25T08:41:03.763 INFO:tasks.workunit.client.0.trial047.stdout: lock_type: shared 2026-02-25T08:41:03.763 INFO:tasks.workunit.client.0.trial047.stdout: tag: 2026-02-25T08:41:03.763 INFO:tasks.workunit.client.0.trial047.stdout: lockers: 2026-02-25T08:41:03.763 INFO:tasks.workunit.client.0.trial047.stdout: client.4204 expiration=2026-02-25T08:43:03.749351+0000 addr=v1:10.20.193.47:0/3358350364 cookie= 2026-02-25T08:41:03.763 INFO:tasks.workunit.client.0.trial047.stdout: client.4207 expiration=0.000000 addr=v1:10.20.193.47:0/1231306443 cookie= 2026-02-25T08:41:03.773 INFO:tasks.workunit.client.0.trial047.stdout:lock: mylock 2026-02-25T08:41:03.774 INFO:tasks.workunit.client.0.trial047.stdout: lock_type: shared 2026-02-25T08:41:03.774 INFO:tasks.workunit.client.0.trial047.stdout: tag: 2026-02-25T08:41:03.774 INFO:tasks.workunit.client.0.trial047.stdout: lockers: 2026-02-25T08:41:03.774 INFO:tasks.workunit.client.0.trial047.stdout: client.4207 expiration=0.000000 addr=v1:10.20.193.47:0/1231306443 cookie= 2026-02-25T08:41:04.442 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestMultiLocking (2862 ms) 2026-02-25T08:41:04.442 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestMeta 2026-02-25T08:41:06.703 INFO:tasks.workunit.client.0.trial047.stdout:lock: mylock 2026-02-25T08:41:06.703 INFO:tasks.workunit.client.0.trial047.stdout: lock_type: shared 2026-02-25T08:41:06.704 INFO:tasks.workunit.client.0.trial047.stdout: tag: 2026-02-25T08:41:06.704 INFO:tasks.workunit.client.0.trial047.stdout: lockers: 2026-02-25T08:41:06.704 INFO:tasks.workunit.client.0.trial047.stdout: client.4213 expiration=0.000000 addr=v1:10.20.193.47:0/3028938625 cookie= 2026-02-25T08:41:06.723 INFO:tasks.workunit.client.0.trial047.stdout:lock: mylock 2026-02-25T08:41:06.723 INFO:tasks.workunit.client.0.trial047.stdout: lock_type: exclusive 2026-02-25T08:41:06.723 INFO:tasks.workunit.client.0.trial047.stdout: tag: new_tag 2026-02-25T08:41:06.723 INFO:tasks.workunit.client.0.trial047.stdout: lockers: 2026-02-25T08:41:06.723 INFO:tasks.workunit.client.0.trial047.stdout: client.4210 expiration=0.000000 addr=v1:10.20.193.47:0/3363088337 cookie= 2026-02-25T08:41:07.453 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestMeta (3010 ms) 2026-02-25T08:41:07.453 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestCookie 2026-02-25T08:41:09.693 INFO:tasks.workunit.client.0.trial047.stdout:lock: mylock 2026-02-25T08:41:09.693 INFO:tasks.workunit.client.0.trial047.stdout: lock_type: exclusive 2026-02-25T08:41:09.693 INFO:tasks.workunit.client.0.trial047.stdout: tag: 2026-02-25T08:41:09.693 INFO:tasks.workunit.client.0.trial047.stdout: lockers: 2026-02-25T08:41:09.713 INFO:tasks.workunit.client.0.trial047.stdout:lock: mylock 2026-02-25T08:41:09.713 INFO:tasks.workunit.client.0.trial047.stdout: lock_type: shared 2026-02-25T08:41:09.713 INFO:tasks.workunit.client.0.trial047.stdout: tag: 2026-02-25T08:41:09.713 INFO:tasks.workunit.client.0.trial047.stdout: lockers: 2026-02-25T08:41:09.713 INFO:tasks.workunit.client.0.trial047.stdout: client.4216 expiration=0.000000 addr=v1:10.20.193.47:0/504681157 cookie= 2026-02-25T08:41:09.713 INFO:tasks.workunit.client.0.trial047.stdout: client.4216 expiration=0.000000 addr=v1:10.20.193.47:0/504681157 cookie=new cookie 2026-02-25T08:41:10.463 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestCookie (3010 ms) 2026-02-25T08:41:10.463 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestMultipleLocks 2026-02-25T08:41:13.474 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestMultipleLocks (3010 ms) 2026-02-25T08:41:13.474 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestLockDuration 2026-02-25T08:41:21.495 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestLockDuration (8021 ms) 2026-02-25T08:41:21.495 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestAssertLocked 2026-02-25T08:41:24.508 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestAssertLocked (3012 ms) 2026-02-25T08:41:24.508 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestSetCookie 2026-02-25T08:41:27.521 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestSetCookie (3012 ms) 2026-02-25T08:41:27.521 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestRenew 2026-02-25T08:41:48.548 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestRenew (21027 ms) 2026-02-25T08:41:48.548 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2026-02-25T08:41:55.568 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7020 ms) 2026-02-25T08:41:55.568 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2026-02-25T08:42:03.589 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (8020 ms) 2026-02-25T08:42:03.589 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2026-02-25T08:42:11.608 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8018 ms) 2026-02-25T08:42:11.608 INFO:tasks.workunit.client.0.trial047.stdout:[----------] 11 tests from ClsLock (70028 ms total) 2026-02-25T08:42:11.608 INFO:tasks.workunit.client.0.trial047.stdout: 2026-02-25T08:42:11.608 INFO:tasks.workunit.client.0.trial047.stdout:[----------] Global test environment tear-down 2026-02-25T08:42:11.608 INFO:tasks.workunit.client.0.trial047.stdout:[==========] 11 tests from 1 test suite ran. (70028 ms total) 2026-02-25T08:42:11.608 INFO:tasks.workunit.client.0.trial047.stdout:[ PASSED ] 11 tests. 2026-02-25T08:42:11.612 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T08:42:11.612 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T08:42:11.680 INFO:tasks.workunit:Running workunits matching cls/test_cls_journal.sh on client.0... 2026-02-25T08:42:11.682 INFO:tasks.workunit:Running workunit cls/test_cls_journal.sh... 2026-02-25T08:42:11.683 DEBUG:teuthology.orchestra.run.trial047:workunit test cls/test_cls_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c32469671c6b0222f786dd36c34eb702f780a11d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_journal.sh 2026-02-25T08:42:11.757 INFO:tasks.workunit.client.0.trial047.stdout:Running main() from gmock_main.cc 2026-02-25T08:42:11.757 INFO:tasks.workunit.client.0.trial047.stdout:[==========] Running 27 tests from 1 test suite. 2026-02-25T08:42:11.758 INFO:tasks.workunit.client.0.trial047.stdout:[----------] Global test environment set-up. 2026-02-25T08:42:11.758 INFO:tasks.workunit.client.0.trial047.stdout:[----------] 27 tests from TestClsJournal 2026-02-25T08:42:13.616 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.Create 2026-02-25T08:42:13.802 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.Create (185 ms) 2026-02-25T08:42:13.802 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.MinimumSet 2026-02-25T08:42:13.854 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.MinimumSet (52 ms) 2026-02-25T08:42:13.854 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.MinimumSetStale 2026-02-25T08:42:13.901 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.MinimumSetStale (46 ms) 2026-02-25T08:42:13.901 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.MinimumSetOrderConstraint 2026-02-25T08:42:13.928 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.MinimumSetOrderConstraint (26 ms) 2026-02-25T08:42:13.928 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ActiveSet 2026-02-25T08:42:13.956 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ActiveSet (28 ms) 2026-02-25T08:42:13.956 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ActiveSetStale 2026-02-25T08:42:13.989 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ActiveSetStale (32 ms) 2026-02-25T08:42:13.989 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.CreateDuplicate 2026-02-25T08:42:14.008 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.CreateDuplicate (19 ms) 2026-02-25T08:42:14.008 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.GetClient 2026-02-25T08:42:14.035 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.GetClient (27 ms) 2026-02-25T08:42:14.035 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientRegister 2026-02-25T08:42:14.062 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientRegister (27 ms) 2026-02-25T08:42:14.062 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientRegisterDuplicate 2026-02-25T08:42:14.091 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientRegisterDuplicate (29 ms) 2026-02-25T08:42:14.092 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientUpdateData 2026-02-25T08:42:14.131 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientUpdateData (39 ms) 2026-02-25T08:42:14.132 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientUpdateState 2026-02-25T08:42:14.171 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientUpdateState (40 ms) 2026-02-25T08:42:14.172 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientUnregister 2026-02-25T08:42:14.202 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientUnregister (30 ms) 2026-02-25T08:42:14.202 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientUnregisterDNE 2026-02-25T08:42:14.245 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientUnregisterDNE (42 ms) 2026-02-25T08:42:14.245 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientUnregisterPruneTags 2026-02-25T08:42:15.464 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientUnregisterPruneTags (1219 ms) 2026-02-25T08:42:15.464 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientCommit 2026-02-25T08:42:15.499 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientCommit (35 ms) 2026-02-25T08:42:15.499 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientCommitInvalid 2026-02-25T08:42:15.529 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientCommitInvalid (29 ms) 2026-02-25T08:42:15.529 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientCommitDNE 2026-02-25T08:42:15.536 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientCommitDNE (7 ms) 2026-02-25T08:42:15.536 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.ClientList 2026-02-25T08:42:17.380 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.ClientList (1843 ms) 2026-02-25T08:42:17.380 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.GetNextTagTid 2026-02-25T08:42:17.429 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.GetNextTagTid (49 ms) 2026-02-25T08:42:17.429 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.TagCreate 2026-02-25T08:42:17.521 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.TagCreate (91 ms) 2026-02-25T08:42:17.521 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.TagCreatePrunesTags 2026-02-25T08:42:17.602 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.TagCreatePrunesTags (81 ms) 2026-02-25T08:42:17.602 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.TagList 2026-02-25T08:42:18.770 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.TagList (1167 ms) 2026-02-25T08:42:18.770 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.GuardAppend 2026-02-25T08:42:18.781 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.GuardAppend (11 ms) 2026-02-25T08:42:18.781 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.GuardAppendDNE 2026-02-25T08:42:18.784 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.GuardAppendDNE (3 ms) 2026-02-25T08:42:18.784 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.GuardAppendOverflow 2026-02-25T08:42:18.802 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.GuardAppendOverflow (17 ms) 2026-02-25T08:42:18.802 INFO:tasks.workunit.client.0.trial047.stdout:[ RUN ] TestClsJournal.Append 2026-02-25T08:42:18.861 INFO:tasks.workunit.client.0.trial047.stdout:padding is not enabled 2026-02-25T08:42:18.861 INFO:tasks.workunit.client.0.trial047.stdout:[ OK ] TestClsJournal.Append (59 ms) 2026-02-25T08:42:18.951 INFO:tasks.workunit.client.0.trial047.stdout:[----------] 27 tests from TestClsJournal (5244 ms total) 2026-02-25T08:42:18.951 INFO:tasks.workunit.client.0.trial047.stdout: 2026-02-25T08:42:18.951 INFO:tasks.workunit.client.0.trial047.stdout:[----------] Global test environment tear-down 2026-02-25T08:42:18.951 INFO:tasks.workunit.client.0.trial047.stdout:[==========] 27 tests from 1 test suite ran. (7193 ms total) 2026-02-25T08:42:18.951 INFO:tasks.workunit.client.0.trial047.stdout:[ PASSED ] 27 tests. 2026-02-25T08:42:18.953 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T08:42:18.954 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T08:42:19.021 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh'] on client.0... 2026-02-25T08:42:19.021 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-25T08:42:19.426 DEBUG:teuthology.parallel:result is None 2026-02-25T08:42:19.426 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T08:42:19.451 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T08:42:19.452 DEBUG:teuthology.orchestra.run.trial047:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T08:42:19.508 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T08:42:19.508 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T08:42:19.523 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:42:19.523 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:42:19.624 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:19.618+0000 7f704db9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:19.627 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:19.623+0000 7f704db9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:19.737 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:42:19.737 INFO:teuthology.orchestra.run.trial047.stderr:dumped all 2026-02-25T08:42:19.746 INFO:teuthology.orchestra.run.trial047.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2026-02-25T08:42:18.946506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":10400,"kb_used_data":10400,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156489568,"statfs":{"total":160255967232,"available":160245317632,"internally_reserved":0,"allocated":10649600,"data_stored":10649600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.054646"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630826+0000","last_change":"2026-02-25T08:39:51.793690+0000","last_active":"2026-02-25T08:42:14.630826+0000","last_peered":"2026-02-25T08:42:14.630826+0000","last_clean":"2026-02-25T08:42:14.630826+0000","last_became_active":"2026-02-25T08:39:49.814925+0000","last_became_peered":"2026-02-25T08:39:49.814925+0000","last_unstale":"2026-02-25T08:42:14.630826+0000","last_undegraded":"2026-02-25T08:42:14.630826+0000","last_fullsized":"2026-02-25T08:42:14.630826+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631244+0000","last_change":"2026-02-25T08:39:51.794622+0000","last_active":"2026-02-25T08:42:14.631244+0000","last_peered":"2026-02-25T08:42:14.631244+0000","last_clean":"2026-02-25T08:42:14.631244+0000","last_became_active":"2026-02-25T08:39:49.808924+0000","last_became_peered":"2026-02-25T08:39:49.808924+0000","last_unstale":"2026-02-25T08:42:14.631244+0000","last_undegraded":"2026-02-25T08:42:14.631244+0000","last_fullsized":"2026-02-25T08:42:14.631244+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630393+0000","last_change":"2026-02-25T08:39:51.797616+0000","last_active":"2026-02-25T08:42:14.630393+0000","last_peered":"2026-02-25T08:42:14.630393+0000","last_clean":"2026-02-25T08:42:14.630393+0000","last_became_active":"2026-02-25T08:39:49.807974+0000","last_became_peered":"2026-02-25T08:39:49.807974+0000","last_unstale":"2026-02-25T08:42:14.630393+0000","last_undegraded":"2026-02-25T08:42:14.630393+0000","last_fullsized":"2026-02-25T08:42:14.630393+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630656+0000","last_change":"2026-02-25T08:39:51.797568+0000","last_active":"2026-02-25T08:42:14.630656+0000","last_peered":"2026-02-25T08:42:14.630656+0000","last_clean":"2026-02-25T08:42:14.630656+0000","last_became_active":"2026-02-25T08:39:49.817503+0000","last_became_peered":"2026-02-25T08:39:49.817503+0000","last_unstale":"2026-02-25T08:42:14.630656+0000","last_undegraded":"2026-02-25T08:42:14.630656+0000","last_fullsized":"2026-02-25T08:42:14.630656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631316+0000","last_change":"2026-02-25T08:39:51.796471+0000","last_active":"2026-02-25T08:42:14.631316+0000","last_peered":"2026-02-25T08:42:14.631316+0000","last_clean":"2026-02-25T08:42:14.631316+0000","last_became_active":"2026-02-25T08:39:49.827668+0000","last_became_peered":"2026-02-25T08:39:49.827668+0000","last_unstale":"2026-02-25T08:42:14.631316+0000","last_undegraded":"2026-02-25T08:42:14.631316+0000","last_fullsized":"2026-02-25T08:42:14.631316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631079+0000","last_change":"2026-02-25T08:39:51.794859+0000","last_active":"2026-02-25T08:42:14.631079+0000","last_peered":"2026-02-25T08:42:14.631079+0000","last_clean":"2026-02-25T08:42:14.631079+0000","last_became_active":"2026-02-25T08:39:49.826833+0000","last_became_peered":"2026-02-25T08:39:49.826833+0000","last_unstale":"2026-02-25T08:42:14.631079+0000","last_undegraded":"2026-02-25T08:42:14.631079+0000","last_fullsized":"2026-02-25T08:42:14.631079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629182+0000","last_change":"2026-02-25T08:39:51.792927+0000","last_active":"2026-02-25T08:42:14.629182+0000","last_peered":"2026-02-25T08:42:14.629182+0000","last_clean":"2026-02-25T08:42:14.629182+0000","last_became_active":"2026-02-25T08:39:49.831241+0000","last_became_peered":"2026-02-25T08:39:49.831241+0000","last_unstale":"2026-02-25T08:42:14.629182+0000","last_undegraded":"2026-02-25T08:42:14.629182+0000","last_fullsized":"2026-02-25T08:42:14.629182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":157,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630071+0000","last_change":"2026-02-25T08:39:51.794944+0000","last_active":"2026-02-25T08:42:14.630071+0000","last_peered":"2026-02-25T08:42:14.630071+0000","last_clean":"2026-02-25T08:42:14.630071+0000","last_became_active":"2026-02-25T08:39:49.810544+0000","last_became_peered":"2026-02-25T08:39:49.810544+0000","last_unstale":"2026-02-25T08:42:14.630071+0000","last_undegraded":"2026-02-25T08:42:14.630071+0000","last_fullsized":"2026-02-25T08:42:14.630071+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":231,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629477+0000","last_change":"2026-02-25T08:39:44.791638+0000","last_active":"2026-02-25T08:42:14.629477+0000","last_peered":"2026-02-25T08:42:14.629477+0000","last_clean":"2026-02-25T08:42:14.629477+0000","last_became_active":"2026-02-25T08:39:44.782576+0000","last_became_peered":"2026-02-25T08:39:44.782576+0000","last_unstale":"2026-02-25T08:42:14.629477+0000","last_undegraded":"2026-02-25T08:42:14.629477+0000","last_fullsized":"2026-02-25T08:42:14.629477+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_clean_scrub_stamp":"2026-02-25T08:39:43.746343+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160245235712,"internally_reserved":0,"allocated":10731520,"data_stored":10731520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640410,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3988,"kb_used_data":3988,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162668,"statfs":{"total":53418655744,"available":53414572032,"internally_reserved":0,"allocated":4083712,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2652,"kb_used_data":2652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164004,"statfs":{"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3760,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162896,"statfs":{"total":53418655744,"available":53414805504,"internally_reserved":0,"allocated":3850240,"data_stored":3850240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53414555648,"internally_reserved":0,"allocated":4100096,"data_stored":4100096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:42:19.748 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:42:19.823 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:19.817+0000 7fe2111dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:19.825 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:19.822+0000 7fe2111dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:19.933 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:42:19.934 INFO:teuthology.orchestra.run.trial047.stderr:dumped all 2026-02-25T08:42:19.944 INFO:teuthology.orchestra.run.trial047.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2026-02-25T08:42:18.946506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":10400,"kb_used_data":10400,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156489568,"statfs":{"total":160255967232,"available":160245317632,"internally_reserved":0,"allocated":10649600,"data_stored":10649600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.054646"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630826+0000","last_change":"2026-02-25T08:39:51.793690+0000","last_active":"2026-02-25T08:42:14.630826+0000","last_peered":"2026-02-25T08:42:14.630826+0000","last_clean":"2026-02-25T08:42:14.630826+0000","last_became_active":"2026-02-25T08:39:49.814925+0000","last_became_peered":"2026-02-25T08:39:49.814925+0000","last_unstale":"2026-02-25T08:42:14.630826+0000","last_undegraded":"2026-02-25T08:42:14.630826+0000","last_fullsized":"2026-02-25T08:42:14.630826+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631244+0000","last_change":"2026-02-25T08:39:51.794622+0000","last_active":"2026-02-25T08:42:14.631244+0000","last_peered":"2026-02-25T08:42:14.631244+0000","last_clean":"2026-02-25T08:42:14.631244+0000","last_became_active":"2026-02-25T08:39:49.808924+0000","last_became_peered":"2026-02-25T08:39:49.808924+0000","last_unstale":"2026-02-25T08:42:14.631244+0000","last_undegraded":"2026-02-25T08:42:14.631244+0000","last_fullsized":"2026-02-25T08:42:14.631244+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630393+0000","last_change":"2026-02-25T08:39:51.797616+0000","last_active":"2026-02-25T08:42:14.630393+0000","last_peered":"2026-02-25T08:42:14.630393+0000","last_clean":"2026-02-25T08:42:14.630393+0000","last_became_active":"2026-02-25T08:39:49.807974+0000","last_became_peered":"2026-02-25T08:39:49.807974+0000","last_unstale":"2026-02-25T08:42:14.630393+0000","last_undegraded":"2026-02-25T08:42:14.630393+0000","last_fullsized":"2026-02-25T08:42:14.630393+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630656+0000","last_change":"2026-02-25T08:39:51.797568+0000","last_active":"2026-02-25T08:42:14.630656+0000","last_peered":"2026-02-25T08:42:14.630656+0000","last_clean":"2026-02-25T08:42:14.630656+0000","last_became_active":"2026-02-25T08:39:49.817503+0000","last_became_peered":"2026-02-25T08:39:49.817503+0000","last_unstale":"2026-02-25T08:42:14.630656+0000","last_undegraded":"2026-02-25T08:42:14.630656+0000","last_fullsized":"2026-02-25T08:42:14.630656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631316+0000","last_change":"2026-02-25T08:39:51.796471+0000","last_active":"2026-02-25T08:42:14.631316+0000","last_peered":"2026-02-25T08:42:14.631316+0000","last_clean":"2026-02-25T08:42:14.631316+0000","last_became_active":"2026-02-25T08:39:49.827668+0000","last_became_peered":"2026-02-25T08:39:49.827668+0000","last_unstale":"2026-02-25T08:42:14.631316+0000","last_undegraded":"2026-02-25T08:42:14.631316+0000","last_fullsized":"2026-02-25T08:42:14.631316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631079+0000","last_change":"2026-02-25T08:39:51.794859+0000","last_active":"2026-02-25T08:42:14.631079+0000","last_peered":"2026-02-25T08:42:14.631079+0000","last_clean":"2026-02-25T08:42:14.631079+0000","last_became_active":"2026-02-25T08:39:49.826833+0000","last_became_peered":"2026-02-25T08:39:49.826833+0000","last_unstale":"2026-02-25T08:42:14.631079+0000","last_undegraded":"2026-02-25T08:42:14.631079+0000","last_fullsized":"2026-02-25T08:42:14.631079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629182+0000","last_change":"2026-02-25T08:39:51.792927+0000","last_active":"2026-02-25T08:42:14.629182+0000","last_peered":"2026-02-25T08:42:14.629182+0000","last_clean":"2026-02-25T08:42:14.629182+0000","last_became_active":"2026-02-25T08:39:49.831241+0000","last_became_peered":"2026-02-25T08:39:49.831241+0000","last_unstale":"2026-02-25T08:42:14.629182+0000","last_undegraded":"2026-02-25T08:42:14.629182+0000","last_fullsized":"2026-02-25T08:42:14.629182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":157,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630071+0000","last_change":"2026-02-25T08:39:51.794944+0000","last_active":"2026-02-25T08:42:14.630071+0000","last_peered":"2026-02-25T08:42:14.630071+0000","last_clean":"2026-02-25T08:42:14.630071+0000","last_became_active":"2026-02-25T08:39:49.810544+0000","last_became_peered":"2026-02-25T08:39:49.810544+0000","last_unstale":"2026-02-25T08:42:14.630071+0000","last_undegraded":"2026-02-25T08:42:14.630071+0000","last_fullsized":"2026-02-25T08:42:14.630071+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":231,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629477+0000","last_change":"2026-02-25T08:39:44.791638+0000","last_active":"2026-02-25T08:42:14.629477+0000","last_peered":"2026-02-25T08:42:14.629477+0000","last_clean":"2026-02-25T08:42:14.629477+0000","last_became_active":"2026-02-25T08:39:44.782576+0000","last_became_peered":"2026-02-25T08:39:44.782576+0000","last_unstale":"2026-02-25T08:42:14.629477+0000","last_undegraded":"2026-02-25T08:42:14.629477+0000","last_fullsized":"2026-02-25T08:42:14.629477+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_clean_scrub_stamp":"2026-02-25T08:39:43.746343+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160245235712,"internally_reserved":0,"allocated":10731520,"data_stored":10731520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640410,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3988,"kb_used_data":3988,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162668,"statfs":{"total":53418655744,"available":53414572032,"internally_reserved":0,"allocated":4083712,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2652,"kb_used_data":2652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164004,"statfs":{"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3760,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162896,"statfs":{"total":53418655744,"available":53414805504,"internally_reserved":0,"allocated":3850240,"data_stored":3850240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53414555648,"internally_reserved":0,"allocated":4100096,"data_stored":4100096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:42:19.945 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:42:19.946 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:42:20.022 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:20.016+0000 7fa99350b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:20.024 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:20.021+0000 7fa99350b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:20.132 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:42:20.133 INFO:teuthology.orchestra.run.trial047.stderr:dumped all 2026-02-25T08:42:20.142 INFO:teuthology.orchestra.run.trial047.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2026-02-25T08:42:18.946506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":10400,"kb_used_data":10400,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156489568,"statfs":{"total":160255967232,"available":160245317632,"internally_reserved":0,"allocated":10649600,"data_stored":10649600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.054646"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630826+0000","last_change":"2026-02-25T08:39:51.793690+0000","last_active":"2026-02-25T08:42:14.630826+0000","last_peered":"2026-02-25T08:42:14.630826+0000","last_clean":"2026-02-25T08:42:14.630826+0000","last_became_active":"2026-02-25T08:39:49.814925+0000","last_became_peered":"2026-02-25T08:39:49.814925+0000","last_unstale":"2026-02-25T08:42:14.630826+0000","last_undegraded":"2026-02-25T08:42:14.630826+0000","last_fullsized":"2026-02-25T08:42:14.630826+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631244+0000","last_change":"2026-02-25T08:39:51.794622+0000","last_active":"2026-02-25T08:42:14.631244+0000","last_peered":"2026-02-25T08:42:14.631244+0000","last_clean":"2026-02-25T08:42:14.631244+0000","last_became_active":"2026-02-25T08:39:49.808924+0000","last_became_peered":"2026-02-25T08:39:49.808924+0000","last_unstale":"2026-02-25T08:42:14.631244+0000","last_undegraded":"2026-02-25T08:42:14.631244+0000","last_fullsized":"2026-02-25T08:42:14.631244+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630393+0000","last_change":"2026-02-25T08:39:51.797616+0000","last_active":"2026-02-25T08:42:14.630393+0000","last_peered":"2026-02-25T08:42:14.630393+0000","last_clean":"2026-02-25T08:42:14.630393+0000","last_became_active":"2026-02-25T08:39:49.807974+0000","last_became_peered":"2026-02-25T08:39:49.807974+0000","last_unstale":"2026-02-25T08:42:14.630393+0000","last_undegraded":"2026-02-25T08:42:14.630393+0000","last_fullsized":"2026-02-25T08:42:14.630393+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630656+0000","last_change":"2026-02-25T08:39:51.797568+0000","last_active":"2026-02-25T08:42:14.630656+0000","last_peered":"2026-02-25T08:42:14.630656+0000","last_clean":"2026-02-25T08:42:14.630656+0000","last_became_active":"2026-02-25T08:39:49.817503+0000","last_became_peered":"2026-02-25T08:39:49.817503+0000","last_unstale":"2026-02-25T08:42:14.630656+0000","last_undegraded":"2026-02-25T08:42:14.630656+0000","last_fullsized":"2026-02-25T08:42:14.630656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631316+0000","last_change":"2026-02-25T08:39:51.796471+0000","last_active":"2026-02-25T08:42:14.631316+0000","last_peered":"2026-02-25T08:42:14.631316+0000","last_clean":"2026-02-25T08:42:14.631316+0000","last_became_active":"2026-02-25T08:39:49.827668+0000","last_became_peered":"2026-02-25T08:39:49.827668+0000","last_unstale":"2026-02-25T08:42:14.631316+0000","last_undegraded":"2026-02-25T08:42:14.631316+0000","last_fullsized":"2026-02-25T08:42:14.631316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631079+0000","last_change":"2026-02-25T08:39:51.794859+0000","last_active":"2026-02-25T08:42:14.631079+0000","last_peered":"2026-02-25T08:42:14.631079+0000","last_clean":"2026-02-25T08:42:14.631079+0000","last_became_active":"2026-02-25T08:39:49.826833+0000","last_became_peered":"2026-02-25T08:39:49.826833+0000","last_unstale":"2026-02-25T08:42:14.631079+0000","last_undegraded":"2026-02-25T08:42:14.631079+0000","last_fullsized":"2026-02-25T08:42:14.631079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629182+0000","last_change":"2026-02-25T08:39:51.792927+0000","last_active":"2026-02-25T08:42:14.629182+0000","last_peered":"2026-02-25T08:42:14.629182+0000","last_clean":"2026-02-25T08:42:14.629182+0000","last_became_active":"2026-02-25T08:39:49.831241+0000","last_became_peered":"2026-02-25T08:39:49.831241+0000","last_unstale":"2026-02-25T08:42:14.629182+0000","last_undegraded":"2026-02-25T08:42:14.629182+0000","last_fullsized":"2026-02-25T08:42:14.629182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":157,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630071+0000","last_change":"2026-02-25T08:39:51.794944+0000","last_active":"2026-02-25T08:42:14.630071+0000","last_peered":"2026-02-25T08:42:14.630071+0000","last_clean":"2026-02-25T08:42:14.630071+0000","last_became_active":"2026-02-25T08:39:49.810544+0000","last_became_peered":"2026-02-25T08:39:49.810544+0000","last_unstale":"2026-02-25T08:42:14.630071+0000","last_undegraded":"2026-02-25T08:42:14.630071+0000","last_fullsized":"2026-02-25T08:42:14.630071+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":231,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629477+0000","last_change":"2026-02-25T08:39:44.791638+0000","last_active":"2026-02-25T08:42:14.629477+0000","last_peered":"2026-02-25T08:42:14.629477+0000","last_clean":"2026-02-25T08:42:14.629477+0000","last_became_active":"2026-02-25T08:39:44.782576+0000","last_became_peered":"2026-02-25T08:39:44.782576+0000","last_unstale":"2026-02-25T08:42:14.629477+0000","last_undegraded":"2026-02-25T08:42:14.629477+0000","last_fullsized":"2026-02-25T08:42:14.629477+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_clean_scrub_stamp":"2026-02-25T08:39:43.746343+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160245235712,"internally_reserved":0,"allocated":10731520,"data_stored":10731520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640410,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3988,"kb_used_data":3988,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162668,"statfs":{"total":53418655744,"available":53414572032,"internally_reserved":0,"allocated":4083712,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2652,"kb_used_data":2652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164004,"statfs":{"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3760,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162896,"statfs":{"total":53418655744,"available":53414805504,"internally_reserved":0,"allocated":3850240,"data_stored":3850240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53414555648,"internally_reserved":0,"allocated":4100096,"data_stored":4100096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:42:20.143 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:42:20.219 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:20.213+0000 7fce8e4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:20.221 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:20.218+0000 7fce8e4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:20.332 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:42:20.332 INFO:teuthology.orchestra.run.trial047.stdout:{"epoch":65,"fsid":"3ac28462-018f-46f5-99a5-a4e0dc86382c","created":"2026-02-25T08:39:32.393981+0000","modified":"2026-02-25T08:42:18.944334+0000","last_up_change":"2026-02-25T08:39:43.746343+0000","last_in_change":"2026-02-25T08:39:34.163595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:39:42.881297+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-25T08:39:48.121029+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4a88c0ef-1d99-4045-9137-dce175291f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6808","nonce":1498602973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6809","nonce":1498602973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6813","nonce":1498602973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6814","nonce":1498602973}]},"public_addr":"10.20.193.47:6808/1498602973","cluster_addr":"10.20.193.47:6809/1498602973","heartbeat_back_addr":"10.20.193.47:6813/1498602973","heartbeat_front_addr":"10.20.193.47:6814/1498602973","state":["exists","up"]},{"osd":1,"uuid":"268df963-9285-40d0-8edb-e48c58f5dcbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6805","nonce":4005732361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6806","nonce":4005732361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6811","nonce":4005732361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6810","nonce":4005732361}]},"public_addr":"10.20.193.47:6805/4005732361","cluster_addr":"10.20.193.47:6806/4005732361","heartbeat_back_addr":"10.20.193.47:6811/4005732361","heartbeat_front_addr":"10.20.193.47:6810/4005732361","state":["exists","up"]},{"osd":2,"uuid":"7f0b7530-efb8-40d7-938e-e70d51050c3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6802","nonce":654768586}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6803","nonce":654768586}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6807","nonce":654768586}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.47:6804","nonce":654768586}]},"public_addr":"10.20.193.47:6802/654768586","cluster_addr":"10.20.193.47:6803/654768586","heartbeat_back_addr":"10.20.193.47:6807/654768586","heartbeat_front_addr":"10.20.193.47:6804/654768586","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:42:21.343 INFO:tasks.ceph:Scrubbing osd.0 2026-02-25T08:42:21.344 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-25T08:42:21.424 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.416+0000 7f30b31b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.426 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.423+0000 7f30b31b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.436 INFO:teuthology.orchestra.run.trial047.stdout:{ 2026-02-25T08:42:21.436 INFO:teuthology.orchestra.run.trial047.stdout: "success": "" 2026-02-25T08:42:21.436 INFO:teuthology.orchestra.run.trial047.stdout:} 2026-02-25T08:42:21.442 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-25T08:42:21.519 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.512+0000 7f0f8c90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.521 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.518+0000 7f0f8c90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.631 INFO:teuthology.orchestra.run.trial047.stderr:instructed osd(s) 0 to deep-scrub 2026-02-25T08:42:21.642 INFO:tasks.ceph:Scrubbing osd.1 2026-02-25T08:42:21.642 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-25T08:42:21.720 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.712+0000 7f51fe3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.722 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.719+0000 7f51fe3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.733 INFO:teuthology.orchestra.run.trial047.stdout:{ 2026-02-25T08:42:21.733 INFO:teuthology.orchestra.run.trial047.stdout: "success": "" 2026-02-25T08:42:21.733 INFO:teuthology.orchestra.run.trial047.stdout:} 2026-02-25T08:42:21.742 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-25T08:42:21.818 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.811+0000 7f5b902cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.820 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:21.816+0000 7f5b902cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.932 INFO:teuthology.orchestra.run.trial047.stderr:instructed osd(s) 1 to deep-scrub 2026-02-25T08:42:21.944 INFO:tasks.ceph:Scrubbing osd.2 2026-02-25T08:42:21.945 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-25T08:42:22.023 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:22.015+0000 7f680665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.025 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:22.022+0000 7f680665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.035 INFO:teuthology.orchestra.run.trial047.stdout:{ 2026-02-25T08:42:22.035 INFO:teuthology.orchestra.run.trial047.stdout: "success": "" 2026-02-25T08:42:22.036 INFO:teuthology.orchestra.run.trial047.stdout:} 2026-02-25T08:42:22.042 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-25T08:42:22.121 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:22.114+0000 7f71bc65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.123 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:22.119+0000 7f71bc65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.232 INFO:teuthology.orchestra.run.trial047.stderr:instructed osd(s) 2 to deep-scrub 2026-02-25T08:42:22.244 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:42:22.321 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:22.315+0000 7f8c4210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.323 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:22.320+0000 7f8c4210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.437 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:42:22.437 INFO:teuthology.orchestra.run.trial047.stderr:dumped all 2026-02-25T08:42:22.448 INFO:teuthology.orchestra.run.trial047.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2026-02-25T08:42:20.893142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":10400,"kb_used_data":10400,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156489568,"statfs":{"total":160255967232,"available":160245317632,"internally_reserved":0,"allocated":10649600,"data_stored":10649600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.281256"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630826+0000","last_change":"2026-02-25T08:39:51.793690+0000","last_active":"2026-02-25T08:42:14.630826+0000","last_peered":"2026-02-25T08:42:14.630826+0000","last_clean":"2026-02-25T08:42:14.630826+0000","last_became_active":"2026-02-25T08:39:49.814925+0000","last_became_peered":"2026-02-25T08:39:49.814925+0000","last_unstale":"2026-02-25T08:42:14.630826+0000","last_undegraded":"2026-02-25T08:42:14.630826+0000","last_fullsized":"2026-02-25T08:42:14.630826+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631244+0000","last_change":"2026-02-25T08:39:51.794622+0000","last_active":"2026-02-25T08:42:14.631244+0000","last_peered":"2026-02-25T08:42:14.631244+0000","last_clean":"2026-02-25T08:42:14.631244+0000","last_became_active":"2026-02-25T08:39:49.808924+0000","last_became_peered":"2026-02-25T08:39:49.808924+0000","last_unstale":"2026-02-25T08:42:14.631244+0000","last_undegraded":"2026-02-25T08:42:14.631244+0000","last_fullsized":"2026-02-25T08:42:14.631244+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630393+0000","last_change":"2026-02-25T08:39:51.797616+0000","last_active":"2026-02-25T08:42:14.630393+0000","last_peered":"2026-02-25T08:42:14.630393+0000","last_clean":"2026-02-25T08:42:14.630393+0000","last_became_active":"2026-02-25T08:39:49.807974+0000","last_became_peered":"2026-02-25T08:39:49.807974+0000","last_unstale":"2026-02-25T08:42:14.630393+0000","last_undegraded":"2026-02-25T08:42:14.630393+0000","last_fullsized":"2026-02-25T08:42:14.630393+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630656+0000","last_change":"2026-02-25T08:39:51.797568+0000","last_active":"2026-02-25T08:42:14.630656+0000","last_peered":"2026-02-25T08:42:14.630656+0000","last_clean":"2026-02-25T08:42:14.630656+0000","last_became_active":"2026-02-25T08:39:49.817503+0000","last_became_peered":"2026-02-25T08:39:49.817503+0000","last_unstale":"2026-02-25T08:42:14.630656+0000","last_undegraded":"2026-02-25T08:42:14.630656+0000","last_fullsized":"2026-02-25T08:42:14.630656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631316+0000","last_change":"2026-02-25T08:39:51.796471+0000","last_active":"2026-02-25T08:42:14.631316+0000","last_peered":"2026-02-25T08:42:14.631316+0000","last_clean":"2026-02-25T08:42:14.631316+0000","last_became_active":"2026-02-25T08:39:49.827668+0000","last_became_peered":"2026-02-25T08:39:49.827668+0000","last_unstale":"2026-02-25T08:42:14.631316+0000","last_undegraded":"2026-02-25T08:42:14.631316+0000","last_fullsized":"2026-02-25T08:42:14.631316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.631079+0000","last_change":"2026-02-25T08:39:51.794859+0000","last_active":"2026-02-25T08:42:14.631079+0000","last_peered":"2026-02-25T08:42:14.631079+0000","last_clean":"2026-02-25T08:42:14.631079+0000","last_became_active":"2026-02-25T08:39:49.826833+0000","last_became_peered":"2026-02-25T08:39:49.826833+0000","last_unstale":"2026-02-25T08:42:14.631079+0000","last_undegraded":"2026-02-25T08:42:14.631079+0000","last_fullsized":"2026-02-25T08:42:14.631079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":156,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629182+0000","last_change":"2026-02-25T08:39:51.792927+0000","last_active":"2026-02-25T08:42:14.629182+0000","last_peered":"2026-02-25T08:42:14.629182+0000","last_clean":"2026-02-25T08:42:14.629182+0000","last_became_active":"2026-02-25T08:39:49.831241+0000","last_became_peered":"2026-02-25T08:39:49.831241+0000","last_unstale":"2026-02-25T08:42:14.629182+0000","last_undegraded":"2026-02-25T08:42:14.629182+0000","last_fullsized":"2026-02-25T08:42:14.629182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":157,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.630071+0000","last_change":"2026-02-25T08:39:51.794944+0000","last_active":"2026-02-25T08:42:14.630071+0000","last_peered":"2026-02-25T08:42:14.630071+0000","last_clean":"2026-02-25T08:42:14.630071+0000","last_became_active":"2026-02-25T08:39:49.810544+0000","last_became_peered":"2026-02-25T08:39:49.810544+0000","last_unstale":"2026-02-25T08:42:14.630071+0000","last_undegraded":"2026-02-25T08:42:14.630071+0000","last_fullsized":"2026-02-25T08:42:14.630071+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:48.764759+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":231,"reported_epoch":64,"state":"active+clean","last_fresh":"2026-02-25T08:42:14.629477+0000","last_change":"2026-02-25T08:39:44.791638+0000","last_active":"2026-02-25T08:42:14.629477+0000","last_peered":"2026-02-25T08:42:14.629477+0000","last_clean":"2026-02-25T08:42:14.629477+0000","last_became_active":"2026-02-25T08:39:44.782576+0000","last_became_peered":"2026-02-25T08:39:44.782576+0000","last_unstale":"2026-02-25T08:42:14.629477+0000","last_undegraded":"2026-02-25T08:42:14.629477+0000","last_fullsized":"2026-02-25T08:42:14.629477+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:39:43.746343+0000","last_clean_scrub_stamp":"2026-02-25T08:39:43.746343+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160245235712,"internally_reserved":0,"allocated":10731520,"data_stored":10731520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640410,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3988,"kb_used_data":3988,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162668,"statfs":{"total":53418655744,"available":53414572032,"internally_reserved":0,"allocated":4083712,"data_stored":4083712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2652,"kb_used_data":2652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164004,"statfs":{"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3760,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52162896,"statfs":{"total":53418655744,"available":53414805504,"internally_reserved":0,"allocated":3850240,"data_stored":3850240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53414555648,"internally_reserved":0,"allocated":4100096,"data_stored":4100096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53414739968,"internally_reserved":0,"allocated":3915776,"data_stored":3915776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53415940096,"internally_reserved":0,"allocated":2715648,"data_stored":2715648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:42:22.456 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.456 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.456 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.457 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.457 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.457 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.457 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.457 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-25T08:39:48.764759+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=48, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.457 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-25T08:39:43.746343+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=39, tm_sec=43, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=42, tm_sec=20, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:42:22.457 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-25T08:42:42.458 DEBUG:teuthology.orchestra.run.trial047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:42:42.536 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:42.530+0000 7f4c31b8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:42.538 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:42.534+0000 7f4c31b8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:42.647 INFO:teuthology.orchestra.run.trial047.stdout: 2026-02-25T08:42:42.647 INFO:teuthology.orchestra.run.trial047.stderr:dumped all 2026-02-25T08:42:42.656 INFO:teuthology.orchestra.run.trial047.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2026-02-25T08:42:40.896174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":5908,"kb_used_data":5908,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156494060,"statfs":{"total":160255967232,"available":160249917440,"internally_reserved":0,"allocated":6049792,"data_stored":6049792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001818"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:21.932797+0000","last_change":"2026-02-25T08:42:21.947020+0000","last_active":"2026-02-25T08:42:21.932797+0000","last_peered":"2026-02-25T08:42:21.932797+0000","last_clean":"2026-02-25T08:42:21.932797+0000","last_became_active":"2026-02-25T08:39:49.814925+0000","last_became_peered":"2026-02-25T08:39:49.814925+0000","last_unstale":"2026-02-25T08:42:21.932797+0000","last_undegraded":"2026-02-25T08:42:21.932797+0000","last_fullsized":"2026-02-25T08:42:21.932797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:42:21.946969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:21.946969+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:21.932813+0000","last_change":"2026-02-25T08:42:21.945937+0000","last_active":"2026-02-25T08:42:21.932813+0000","last_peered":"2026-02-25T08:42:21.932813+0000","last_clean":"2026-02-25T08:42:21.932813+0000","last_became_active":"2026-02-25T08:39:49.808924+0000","last_became_peered":"2026-02-25T08:39:49.808924+0000","last_unstale":"2026-02-25T08:42:21.932813+0000","last_undegraded":"2026-02-25T08:42:21.932813+0000","last_fullsized":"2026-02-25T08:42:21.932813+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:42:21.945879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:21.945879+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:21.932916+0000","last_change":"2026-02-25T08:42:21.946542+0000","last_active":"2026-02-25T08:42:21.932916+0000","last_peered":"2026-02-25T08:42:21.932916+0000","last_clean":"2026-02-25T08:42:21.932916+0000","last_became_active":"2026-02-25T08:39:49.807974+0000","last_became_peered":"2026-02-25T08:39:49.807974+0000","last_unstale":"2026-02-25T08:42:21.932916+0000","last_undegraded":"2026-02-25T08:42:21.932916+0000","last_fullsized":"2026-02-25T08:42:21.932916+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:42:21.946483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:21.946483+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:21.932291+0000","last_change":"2026-02-25T08:42:21.944952+0000","last_active":"2026-02-25T08:42:21.932291+0000","last_peered":"2026-02-25T08:42:21.932291+0000","last_clean":"2026-02-25T08:42:21.932291+0000","last_became_active":"2026-02-25T08:39:49.817503+0000","last_became_peered":"2026-02-25T08:39:49.817503+0000","last_unstale":"2026-02-25T08:42:21.932291+0000","last_undegraded":"2026-02-25T08:42:21.932291+0000","last_fullsized":"2026-02-25T08:42:21.932291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:42:21.944896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:21.944896+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":163,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:21.630488+0000","last_change":"2026-02-25T08:42:21.640204+0000","last_active":"2026-02-25T08:42:21.630488+0000","last_peered":"2026-02-25T08:42:21.630488+0000","last_clean":"2026-02-25T08:42:21.630488+0000","last_became_active":"2026-02-25T08:39:49.827668+0000","last_became_peered":"2026-02-25T08:39:49.827668+0000","last_unstale":"2026-02-25T08:42:21.630488+0000","last_undegraded":"2026-02-25T08:42:21.630488+0000","last_fullsized":"2026-02-25T08:42:21.630488+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'2","last_scrub_stamp":"2026-02-25T08:42:21.640134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:21.640134+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:22.232189+0000","last_change":"2026-02-25T08:42:22.240027+0000","last_active":"2026-02-25T08:42:22.232189+0000","last_peered":"2026-02-25T08:42:22.232189+0000","last_clean":"2026-02-25T08:42:22.232189+0000","last_became_active":"2026-02-25T08:39:49.826833+0000","last_became_peered":"2026-02-25T08:39:49.826833+0000","last_unstale":"2026-02-25T08:42:22.232189+0000","last_undegraded":"2026-02-25T08:42:22.232189+0000","last_fullsized":"2026-02-25T08:42:22.232189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:42:22.239947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:22.239947+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:22.231667+0000","last_change":"2026-02-25T08:42:22.239414+0000","last_active":"2026-02-25T08:42:22.231667+0000","last_peered":"2026-02-25T08:42:22.231667+0000","last_clean":"2026-02-25T08:42:22.231667+0000","last_became_active":"2026-02-25T08:39:49.831241+0000","last_became_peered":"2026-02-25T08:39:49.831241+0000","last_unstale":"2026-02-25T08:42:22.231667+0000","last_undegraded":"2026-02-25T08:42:22.231667+0000","last_fullsized":"2026-02-25T08:42:22.231667+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:42:22.239344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:22.239344+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":162,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:21.931773+0000","last_change":"2026-02-25T08:42:21.943090+0000","last_active":"2026-02-25T08:42:21.931773+0000","last_peered":"2026-02-25T08:42:21.931773+0000","last_clean":"2026-02-25T08:42:21.931773+0000","last_became_active":"2026-02-25T08:39:49.810544+0000","last_became_peered":"2026-02-25T08:39:49.810544+0000","last_unstale":"2026-02-25T08:42:21.931773+0000","last_undegraded":"2026-02-25T08:42:21.931773+0000","last_fullsized":"2026-02-25T08:42:21.931773+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'1","last_scrub_stamp":"2026-02-25T08:42:21.942998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:21.942998+0000","last_clean_scrub_stamp":"2026-02-25T08:39:48.764759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":236,"reported_epoch":65,"state":"active+clean","last_fresh":"2026-02-25T08:42:21.931337+0000","last_change":"2026-02-25T08:42:21.956152+0000","last_active":"2026-02-25T08:42:21.931337+0000","last_peered":"2026-02-25T08:42:21.931337+0000","last_clean":"2026-02-25T08:42:21.931337+0000","last_became_active":"2026-02-25T08:39:44.782576+0000","last_became_peered":"2026-02-25T08:39:44.782576+0000","last_unstale":"2026-02-25T08:42:21.931337+0000","last_undegraded":"2026-02-25T08:42:21.931337+0000","last_fullsized":"2026-02-25T08:42:21.931337+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2026-02-25T08:42:21.956064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:42:21.956064+0000","last_clean_scrub_stamp":"2026-02-25T08:39:43.746343+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160249917440,"internally_reserved":0,"allocated":6049792,"data_stored":6049792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53416169472,"internally_reserved":0,"allocated":2486272,"data_stored":2486272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640430,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2416,"kb_used_data":2416,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164240,"statfs":{"total":53418655744,"available":53416181760,"internally_reserved":0,"allocated":2473984,"data_stored":2473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1064,"kb_used_data":1064,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165592,"statfs":{"total":53418655744,"available":53417566208,"internally_reserved":0,"allocated":1089536,"data_stored":1089536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2428,"kb_used_data":2428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164228,"statfs":{"total":53418655744,"available":53416169472,"internally_reserved":0,"allocated":2486272,"data_stored":2486272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53416169472,"internally_reserved":0,"allocated":2486272,"data_stored":2486272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53416181760,"internally_reserved":0,"allocated":2473984,"data_stored":2473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53416169472,"internally_reserved":0,"allocated":2486272,"data_stored":2486272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417566208,"internally_reserved":0,"allocated":1089536,"data_stored":1089536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:42:42.658 DEBUG:teuthology.orchestra.run.trial047:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-25T08:42:42.733 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:42.725+0000 7f0e9241a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:42.735 INFO:teuthology.orchestra.run.trial047.stderr:2026-02-25T08:42:42.731+0000 7f0e9241a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:42.862 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T08:42:42.862 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T08:42:42.862 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T08:42:42.862 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:42:42.930 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T08:42:42.930 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T08:42:42.930 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:42:42.996 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T08:42:42.996 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T08:42:42.996 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:42:43.072 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T08:42:43.072 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T08:42:43.072 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T08:42:43.072 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:42:43.107 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T08:42:43.107 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T08:42:43.108 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T08:42:43.108 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:42:43.158 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T08:42:43.158 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T08:42:43.158 DEBUG:teuthology.orchestra.run.trial047:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-25T08:42:43.223 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:42:43.224 DEBUG:teuthology.orchestra.run.trial047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T08:42:43.299 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:42:43.300 DEBUG:teuthology.orchestra.run.trial047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T08:42:43.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:42:43.373 DEBUG:teuthology.orchestra.run.trial047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T08:42:43.441 INFO:tasks.ceph:Archiving mon data... 2026-02-25T08:42:43.442 DEBUG:teuthology.misc:Transferring archived files from trial047:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69030/data/mon.a.tgz 2026-02-25T08:42:43.443 DEBUG:teuthology.orchestra.run.trial047:> mktemp 2026-02-25T08:42:43.497 INFO:teuthology.orchestra.run.trial047.stdout:/tmp/tmp.WmqUJY24yo 2026-02-25T08:42:43.498 DEBUG:teuthology.orchestra.run.trial047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WmqUJY24yo 2026-02-25T08:42:43.589 DEBUG:teuthology.orchestra.run.trial047:> sudo chmod 0666 /tmp/tmp.WmqUJY24yo 2026-02-25T08:42:43.671 DEBUG:teuthology.orchestra.remote:trial047:/tmp/tmp.WmqUJY24yo is 231KB 2026-02-25T08:42:43.764 DEBUG:teuthology.orchestra.run.trial047:> rm -fr /tmp/tmp.WmqUJY24yo 2026-02-25T08:42:43.781 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T08:42:43.781 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:42:43.902 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:42:43.902 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T08:42:43.907 DEBUG:teuthology.misc:Transferring archived files from trial047:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69030/remote/trial047/crash 2026-02-25T08:42:43.907 DEBUG:teuthology.orchestra.run.trial047:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:42:43.966 INFO:tasks.ceph:Compressing logs... 2026-02-25T08:42:43.966 DEBUG:teuthology.orchestra.run.trial047:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:42:44.030 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T08:42:44.031 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T08:42:44.031 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41018.log 2026-02-25T08:42:44.031 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40721.log 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41018.log.gz 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41898.log 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.40721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40721.log.gz 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43110.log 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41898.log.gz 2026-02-25T08:42:44.046 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42094.log 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43110.log.gz 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42722.log 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42094.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42094.log.gz 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41805.log 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42722.log.gz 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42914.log 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41805.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41805.log.gz 2026-02-25T08:42:44.047 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41657.log 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42914.log.gz 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41803.log 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41657.log.gz 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43208.log 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41803.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41803.log.gz 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41925.log 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43355.log 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43208.log.gz 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42441.log 2026-02-25T08:42:44.048 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41925.log.gz 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43451.log 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43355.log.gz 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42441.log.gz 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43451.log.gz 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43257.log 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.40490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40768.log 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2026-02-25T08:42:44.049 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43257.log.gz 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.40768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41248.log 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40768.log.gz 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.40987.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41248.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41248.log.gz 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41457.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39104.log 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40395.log 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr: 87.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph.tmp-client.admin.39104.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T08:42:44.050 INFO:teuthology.orchestra.run.trial047.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39104.log.gz 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.40395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41318.log 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40395.log.gz 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41343.log 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41318.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41318.log.gz 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40674.log 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41343.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gz 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41658.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43404.log 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.40674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43159.log 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40674.log.gz 2026-02-25T08:42:44.051 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42865.log 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43404.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43404.log.gz 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43159.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43159.log.gz 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41804.log 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42865.log.gz 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41804.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41804.log.gz 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41506.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43306.log 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz 2026-02-25T08:42:44.052 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.40537.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz/var/log/ceph/ceph-client.admin.43306.log: 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42368.log 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43306.log.gz 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41408.log 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43012.log 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.42368.log: 85.2% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42368.log.gz 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.41408.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41408.log.gz 2026-02-25T08:42:44.053 INFO:teuthology.orchestra.run.trial047.stderr:/var/log/ceph/ceph-client.admin.43012.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43012.log.gz 2026-02-25T08:42:44.055 INFO:teuthology.orchestra.run.trial047.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T08:42:44.088 INFO:teuthology.orchestra.run.trial047.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T08:42:45.171 INFO:teuthology.orchestra.run.trial047.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T08:42:45.566 INFO:teuthology.orchestra.run.trial047.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T08:42:46.104 INFO:teuthology.orchestra.run.trial047.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T08:42:46.106 INFO:teuthology.orchestra.run.trial047.stderr: 2026-02-25T08:42:46.106 INFO:teuthology.orchestra.run.trial047.stderr:real 0m2.084s 2026-02-25T08:42:46.106 INFO:teuthology.orchestra.run.trial047.stderr:user 0m4.562s 2026-02-25T08:42:46.106 INFO:teuthology.orchestra.run.trial047.stderr:sys 0m0.289s 2026-02-25T08:42:46.107 INFO:tasks.ceph:Archiving logs... 2026-02-25T08:42:46.107 DEBUG:teuthology.misc:Transferring archived files from trial047:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69030/remote/trial047/log 2026-02-25T08:42:46.109 DEBUG:teuthology.orchestra.run.trial047:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:42:46.752 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:42:50.226 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:42:50.243 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T08:42:50.243 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:42:50.270 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T08:42:50.286 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T08:42:50.286 DEBUG:teuthology.orchestra.run.trial047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:42:50.325 INFO:teuthology.orchestra.run.trial047.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:42:50.429 INFO:teuthology.orchestra.run.trial047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:42:50.429 INFO:teuthology.orchestra.run.trial047.stdout:=============================================================================== 2026-02-25T08:42:50.429 INFO:teuthology.orchestra.run.trial047.stdout:^- 24-111-79-186-dynamic.mi> 2 8 377 123 +4133us[+4006us] +/- 88ms 2026-02-25T08:42:50.429 INFO:teuthology.orchestra.run.trial047.stdout:^- t2.time.bf1.yahoo.com 2 7 377 115 -36ms[ -37ms] +/- 42ms 2026-02-25T08:42:50.429 INFO:teuthology.orchestra.run.trial047.stdout:^- 172-104-28-175.ip.linode> 2 7 377 61 -7436us[-7567us] +/- 60ms 2026-02-25T08:42:50.429 INFO:teuthology.orchestra.run.trial047.stdout:^* time.gslb.gravinoc.com 2 8 377 59 -435us[ -566us] +/- 38ms 2026-02-25T08:42:50.431 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T08:42:50.446 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T08:42:50.447 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T08:42:50.464 DEBUG:teuthology.orchestra.run.trial047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:42:50.526 DEBUG:teuthology.orchestra.run.trial047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:42:50.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:42:50.553 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T08:42:50.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T08:42:50.589 INFO:teuthology.task.internal:Duration was 2356.296782 seconds 2026-02-25T08:42:50.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T08:42:50.605 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T08:42:50.606 DEBUG:teuthology.orchestra.run.trial047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:42:50.641 INFO:teuthology.orchestra.run.trial047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:42:51.032 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T08:42:51.032 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial047.front.sepia.ceph.com 2026-02-25T08:42:51.032 DEBUG:teuthology.orchestra.run.trial047:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:42:51.090 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T08:42:51.090 DEBUG:teuthology.orchestra.run.trial047:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:42:51.193 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T08:42:51.193 DEBUG:teuthology.orchestra.run.trial047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:42:51.216 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:42:51.217 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:42:51.218 INFO:teuthology.orchestra.run.trial047.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:42:51.218 INFO:teuthology.orchestra.run.trial047.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:42:51.218 INFO:teuthology.orchestra.run.trial047.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:42:51.228 INFO:teuthology.orchestra.run.trial047.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:42:51.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T08:42:51.247 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T08:42:51.247 DEBUG:teuthology.orchestra.run.trial047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:42:51.294 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T08:42:51.311 DEBUG:teuthology.orchestra.run.trial047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:42:51.358 INFO:teuthology.orchestra.run.trial047.stdout:kernel.core_pattern = core 2026-02-25T08:42:51.376 DEBUG:teuthology.orchestra.run.trial047:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:42:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:42:51.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T08:42:51.449 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T08:42:51.450 DEBUG:teuthology.misc:Transferring archived files from trial047:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69030/remote/trial047 2026-02-25T08:42:51.450 DEBUG:teuthology.orchestra.run.trial047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:42:51.507 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T08:42:51.507 DEBUG:teuthology.orchestra.run.trial047:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:42:51.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T08:42:51.574 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T08:42:51.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T08:42:51.591 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T08:42:51.592 DEBUG:teuthology.orchestra.run.trial047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:42:51.611 INFO:teuthology.orchestra.run.trial047.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:42 /home/ubuntu/cephtest 2026-02-25T08:42:51.612 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T08:42:51.630 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T08:42:51.668 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} tasks/rbd_cls_tests} duration: 2356.2967824935913 flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com success: true 2026-02-25T08:42:51.669 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:42:52.734 INFO:teuthology.run:pass