2026-02-25T08:03:21.440 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T08:03:21.455 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:03:21.537 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072 branch: wip-logstoremanager description: crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/segmented$/{crimson_seastore_segmented} rados} email: null first_in_suite: false flavor: debug job_id: '69072' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-logstoremanager ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 30 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd pool default crimson: true osd: bluestore zero block detection: true crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: seastore osd op queue: debug_random osd op queue cut off: debug_random seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: c32469671c6b0222f786dd36c34eb702f780a11d owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 869 sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: c32469671c6b0222f786dd36c34eb702f780a11d targets: trial134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJu4mEylc8n2tExh8UuvaV6MfNEUJLoPm8uE+pYer8rpGwDPLHkpyQMHF/LGf6RIm5SZzYyeWSxV1jcdMIBZ8ng= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: global: null osd: crimson cpu num: 2 osd min pg log entries: 5 log-ignorelist: - but it is still running - overall HEALTH_ pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: client.0: - cephtool/test_crimson_dump_metrics.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_07:55:35 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T08:03:21.537 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa; will attempt to use it 2026-02-25T08:03:21.537 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks 2026-02-25T08:03:21.537 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T08:03:21.539 INFO:teuthology.task.internal:Checking packages... 2026-02-25T08:03:21.539 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '6aeb6975bc9a4790a399c9cc70dba9635d2a7819' 2026-02-25T08:03:21.539 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T08:03:21.539 INFO:teuthology.packaging:ref: None 2026-02-25T08:03:21.539 INFO:teuthology.packaging:tag: None 2026-02-25T08:03:21.539 INFO:teuthology.packaging:branch: wip-logstoremanager 2026-02-25T08:03:21.539 INFO:teuthology.packaging:sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:03:21.540 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-logstoremanager 2026-02-25T08:03:21.657 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5338.g6aeb6975 2026-02-25T08:03:21.658 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T08:03:21.671 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T08:03:21.671 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T08:03:21.685 INFO:teuthology.task.internal:Saving configuration 2026-02-25T08:03:21.696 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T08:03:21.708 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T08:03:21.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial134.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:45.957192', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJu4mEylc8n2tExh8UuvaV6MfNEUJLoPm8uE+pYer8rpGwDPLHkpyQMHF/LGf6RIm5SZzYyeWSxV1jcdMIBZ8ng='} 2026-02-25T08:03:21.786 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T08:03:21.798 INFO:teuthology.task.internal:roles: ubuntu@trial134.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-25T08:03:21.798 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T08:03:21.882 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x70379b629c60>, signals=[15]) 2026-02-25T08:03:21.882 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T08:03:21.892 INFO:teuthology.task.internal:Opening connections... 2026-02-25T08:03:21.893 DEBUG:teuthology.task.internal:connecting to ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:03:21.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:21.958 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T08:03:21.970 DEBUG:teuthology.orchestra.run.trial134:> uname -m 2026-02-25T08:03:22.037 INFO:teuthology.orchestra.run.trial134.stdout:x86_64 2026-02-25T08:03:22.037 DEBUG:teuthology.orchestra.run.trial134:> cat /etc/os-release 2026-02-25T08:03:22.093 INFO:teuthology.orchestra.run.trial134.stdout:NAME="CentOS Stream" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:VERSION="9" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:ID="centos" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:VERSION_ID="9" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:ANSI_COLOR="0;31" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:03:22.094 INFO:teuthology.orchestra.run.trial134.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:03:22.095 INFO:teuthology.orchestra.run.trial134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:03:22.095 INFO:teuthology.orchestra.run.trial134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:03:22.096 INFO:teuthology.lock.ops:Updating trial134.front.sepia.ceph.com on lock server 2026-02-25T08:03:22.190 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T08:03:22.205 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T08:03:22.218 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T08:03:22.218 DEBUG:teuthology.orchestra.run.trial134:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:03:22.233 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T08:03:22.244 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T08:03:22.244 DEBUG:teuthology.orchestra.run.trial134:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:03:22.289 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T08:03:22.362 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T08:03:22.381 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T08:03:22.381 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T08:03:22.381 DEBUG:teuthology.orchestra.run.trial134:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:03:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:03:22.396 DEBUG:teuthology.orchestra.run.trial134:> uname -r 2026-02-25T08:03:22.451 INFO:teuthology.orchestra.run.trial134.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:03:22.452 INFO:teuthology.task.kernel:Running kernel on trial134: 5.14.0-681.el9.x86_64 2026-02-25T08:03:22.452 DEBUG:teuthology.orchestra.run.trial134:> sudo yum install -y kernel 2026-02-25T08:03:24.202 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.9 MB 00:01 2026-02-25T08:03:25.110 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - AppStream 71 MB/s | 27 MB 00:00 2026-02-25T08:03:28.773 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 8.0 MB 00:01 2026-02-25T08:03:30.849 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 20 kB 00:01 2026-02-25T08:03:31.954 INFO:teuthology.orchestra.run.trial134.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2026-02-25T08:03:35.273 INFO:teuthology.orchestra.run.trial134.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T08:03:36.279 INFO:teuthology.orchestra.run.trial134.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:03:36.280 INFO:teuthology.orchestra.run.trial134.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:03:36.300 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:03:36.300 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:03:36.300 INFO:teuthology.orchestra.run.trial134.stdout: Package Architecture Version Repository Size 2026-02-25T08:03:36.300 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout:Install 4 Packages 2026-02-25T08:03:36.301 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:03:36.302 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 89 M 2026-02-25T08:03:36.302 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 135 M 2026-02-25T08:03:36.302 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:03:36.522 INFO:teuthology.orchestra.run.trial134.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 6.3 MB/s | 901 kB 00:00 2026-02-25T08:03:37.203 INFO:teuthology.orchestra.run.trial134.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 21 MB/s | 17 MB 00:00 2026-02-25T08:03:37.880 INFO:teuthology.orchestra.run.trial134.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 23 MB/s | 31 MB 00:01 2026-02-25T08:03:38.553 INFO:teuthology.orchestra.run.trial134.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 18 MB/s | 40 MB 00:02 2026-02-25T08:03:38.553 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:03:38.553 INFO:teuthology.orchestra.run.trial134.stdout:Total 39 MB/s | 89 MB 00:02 2026-02-25T08:03:38.744 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:03:38.948 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:03:38.948 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:03:39.331 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:03:39.332 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:03:40.457 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:03:40.762 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:03:40.784 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:43.346 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:43.378 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:46.413 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:46.420 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:49.468 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:06.908 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:06.932 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:07.121 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:07.121 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:07.121 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:07.121 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:07.178 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:07.178 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:04:07.178 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:04:07.178 INFO:teuthology.orchestra.run.trial134.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.178 INFO:teuthology.orchestra.run.trial134.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.178 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.178 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.179 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:04:07.179 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:04:07.239 DEBUG:teuthology.orchestra.run.trial134:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:07.270 INFO:teuthology.orchestra.run.trial134.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.271 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:07.271 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:04:07.271 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:04:07.272 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:04:07.272 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T08:04:07.272 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial134.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:04:07.272 DEBUG:teuthology.orchestra.run.trial134:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:07.336 INFO:teuthology.orchestra.run.trial134.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.337 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:07.337 DEBUG:teuthology.orchestra.run.trial134:> sudo rpm -qi grub2-tools 2026-02-25T08:04:07.405 INFO:teuthology.orchestra.run.trial134.stdout:Name : grub2-tools 2026-02-25T08:04:07.405 INFO:teuthology.orchestra.run.trial134.stdout:Epoch : 1 2026-02-25T08:04:07.405 INFO:teuthology.orchestra.run.trial134.stdout:Version : 2.06 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Release : 120.el9 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Architecture: x86_64 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Group : Unspecified 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Size : 9108611 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:License : GPLv3+ 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Packager : builder@centos.org 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Vendor : CentOS 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Summary : Support tools for GRUB. 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:Description : 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:hardware devices. 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:04:07.406 INFO:teuthology.orchestra.run.trial134.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:04:07.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:04:07.407 INFO:teuthology.task.kernel:Updating grub on trial134 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:04:07.407 DEBUG:teuthology.orchestra.run.trial134:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:04:07.470 INFO:teuthology.orchestra.run.trial134.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:04:07.471 DEBUG:teuthology.orchestra.run.trial134:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:04:07.533 INFO:teuthology.orchestra.run.trial134.stdout:/sbin/grubby 2026-02-25T08:04:07.534 INFO:teuthology.orchestra.run.trial134.stdout:yes 2026-02-25T08:04:07.535 DEBUG:teuthology.orchestra.run.trial134:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:04:07.598 INFO:teuthology.orchestra.run.trial134.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:04:07.599 DEBUG:teuthology.orchestra.run.trial134:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.671 DEBUG:teuthology.orchestra.run.trial134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:04:07.880 INFO:teuthology.orchestra.run.trial134.stderr:Generating grub configuration file ... 2026-02-25T08:04:07.949 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:04:07.956 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:04:08.204 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:04:08.211 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:04:08.246 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:04:08.252 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:04:08.283 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:04:08.289 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:04:08.313 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:04:08.320 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:04:08.406 INFO:teuthology.orchestra.run.trial134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:04:08.414 INFO:teuthology.orchestra.run.trial134.stderr:done 2026-02-25T08:04:08.417 DEBUG:teuthology.orchestra.run.trial134:> test -d /sys/firmware/efi 2026-02-25T08:04:08.433 DEBUG:teuthology.orchestra.run.trial134:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:04:08.498 INFO:teuthology.orchestra.run.trial134.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:04:08.499 INFO:teuthology.orchestra.run.trial134.stdout:/boot/efi/EFI/centos 2026-02-25T08:04:08.500 DEBUG:teuthology.orchestra.run.trial134:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:04:08.565 INFO:teuthology.orchestra.run.trial134.stdout:no 2026-02-25T08:04:08.565 DEBUG:teuthology.orchestra.run.trial134:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:04:08.629 INFO:teuthology.orchestra.run.trial134.stdout:yes 2026-02-25T08:04:08.630 DEBUG:teuthology.orchestra.run.trial134:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:04:08.693 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial134 2026-02-25T08:04:08.693 DEBUG:teuthology.orchestra.run.trial134:> sudo bash -c ' 2026-02-25T08:04:08.693 DEBUG:teuthology.orchestra.run.trial134:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:04:08.693 DEBUG:teuthology.orchestra.run.trial134:> ' 2026-02-25T08:04:38.736 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T08:04:38.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-25T08:04:38.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:04:52.992 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.134 2026-02-25T08:05:01.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-25T08:05:01.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:37.894 DEBUG:teuthology.orchestra.run.trial134:> true 2026-02-25T08:05:38.047 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-25T08:05:38.047 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:05:38.047 DEBUG:teuthology.orchestra.run.trial134:> uname -r 2026-02-25T08:05:38.102 INFO:teuthology.orchestra.run.trial134.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:05:38.102 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:05:38.103 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:05:38.103 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:05:39.104 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T08:05:39.104 DEBUG:teuthology.orchestra.run.trial134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:05:39.133 INFO:teuthology.orchestra.run.trial134.stdout:ttyS1 2026-02-25T08:05:39.158 DEBUG:teuthology.parallel:result is None 2026-02-25T08:05:39.158 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T08:05:39.170 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T08:05:39.170 DEBUG:teuthology.orchestra.run.trial134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:05:39.215 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T08:05:39.226 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T08:05:39.263 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T08:05:39.263 DEBUG:teuthology.orchestra.run.trial134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:05:39.281 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T08:05:39.292 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T08:05:39.293 DEBUG:teuthology.orchestra.run.trial134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:05:39.343 INFO:teuthology.orchestra.run.trial134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:39.352 INFO:teuthology.orchestra.run.trial134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:39.353 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T08:05:39.366 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T08:05:39.366 DEBUG:teuthology.orchestra.run.trial134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:05:39.417 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T08:05:39.432 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T08:05:39.432 DEBUG:teuthology.orchestra.run.trial134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:05:39.473 DEBUG:teuthology.orchestra.run.trial134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:05:39.604 DEBUG:teuthology.orchestra.run.trial134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:05:39.628 DEBUG:teuthology.orchestra.run.trial134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:05:39.747 DEBUG:teuthology.orchestra.run.trial134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:05:39.771 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:05:39.771 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:05:39.833 DEBUG:teuthology.orchestra.run.trial134:> sudo service rsyslog restart 2026-02-25T08:05:39.901 INFO:teuthology.orchestra.run.trial134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:05:40.268 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T08:05:40.282 INFO:teuthology.task.internal:Starting timer... 2026-02-25T08:05:40.282 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T08:05:40.300 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T08:05:40.313 DEBUG:teuthology.orchestra.run.trial134:> sudo service auditd rotate 2026-02-25T08:05:40.340 INFO:teuthology.orchestra.run.trial134.stdout:Rotating logs: 2026-02-25T08:05:40.341 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T08:05:40.341 DEBUG:teuthology.orchestra.run.trial134:> /usr/sbin/getenforce 2026-02-25T08:05:40.397 INFO:teuthology.orchestra.run.trial134.stdout:Permissive 2026-02-25T08:05:40.397 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial134.front.sepia.ceph.com': 'permissive'} 2026-02-25T08:05:40.397 DEBUG:teuthology.orchestra.run.trial134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:05:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:05:40.461 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T08:05:40.462 DEBUG:teuthology.orchestra.run.trial134:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:05:40.524 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T08:05:40.537 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T08:05:40.544 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T08:05:40.544 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T08:08:08.913 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial134.front.sepia.ceph.com')] 2026-02-25T08:08:08.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-25T08:08:08.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:08:08.985 DEBUG:teuthology.orchestra.run.trial134:> true 2026-02-25T08:08:09.062 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-25T08:08:09.062 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T08:08:09.077 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T08:08:09.077 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:08:09.078 DEBUG:teuthology.orchestra.run.trial134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:08:09.129 INFO:teuthology.orchestra.run.trial134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:08:09.142 INFO:teuthology.orchestra.run.trial134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:08:09.162 INFO:teuthology.orchestra.run.trial134.stderr:sudo: ntpd: command not found 2026-02-25T08:08:09.172 INFO:teuthology.orchestra.run.trial134.stdout:506 Cannot talk to daemon 2026-02-25T08:08:09.183 INFO:teuthology.orchestra.run.trial134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:08:09.195 INFO:teuthology.orchestra.run.trial134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:08:09.242 INFO:teuthology.orchestra.run.trial134.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:08:09.262 INFO:teuthology.orchestra.run.trial134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:08:09.262 INFO:teuthology.orchestra.run.trial134.stdout:=============================================================================== 2026-02-25T08:08:09.262 INFO:teuthology.orchestra.run.trial134.stdout:^? ntp1.wiktel.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:09.262 INFO:teuthology.orchestra.run.trial134.stdout:^? ntp.shastacoe.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:09.262 INFO:teuthology.orchestra.run.trial134.stdout:^? arm1.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:09.262 INFO:teuthology.orchestra.run.trial134.stdout:^? pool-138-89-14-60.mad.ea> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:09.262 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:08:09.273 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:08:09.273 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:08:09.273 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:08:09.273 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:08:09.276 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:08:09.276 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-25T08:08:09.277 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:08:09.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:08:09.400 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:08:09.400 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:08:09.732 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:08:09.732 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:08:09.732 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:08:09.758 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:08:09.758 DEBUG:teuthology.orchestra.run.trial134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:08:09.822 DEBUG:teuthology.orchestra.run.trial134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:08:09.895 DEBUG:teuthology.orchestra.run.trial134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:08:09.966 INFO:teuthology.orchestra.run.trial134.stdout:check_obsoletes = 1 2026-02-25T08:08:09.967 DEBUG:teuthology.orchestra.run.trial134:> sudo yum clean all 2026-02-25T08:08:10.117 INFO:teuthology.orchestra.run.trial134.stdout:41 files removed 2026-02-25T08:08:10.131 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-radosgw 2026-02-25T08:08:10.982 INFO:teuthology.orchestra.run.trial134.stdout:ceph packages for x86_64 134 kB/s | 94 kB 00:00 2026-02-25T08:08:11.655 INFO:teuthology.orchestra.run.trial134.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-25T08:08:12.258 INFO:teuthology.orchestra.run.trial134.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-25T08:08:12.524 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - BaseOS 37 MB/s | 8.9 MB 00:00 2026-02-25T08:08:15.694 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-25T08:08:19.028 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 8.0 MB 00:00 2026-02-25T08:08:20.479 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 20 kB 00:00 2026-02-25T08:08:21.461 INFO:teuthology.orchestra.run.trial134.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2026-02-25T08:08:24.661 INFO:teuthology.orchestra.run.trial134.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T08:08:25.881 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repository Size 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout:Upgrading: 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-25T08:08:25.883 INFO:teuthology.orchestra.run.trial134.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:08:25.884 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout:Installing weak dependencies: 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout:Install 35 Packages 2026-02-25T08:08:25.885 INFO:teuthology.orchestra.run.trial134.stdout:Upgrade 2 Packages 2026-02-25T08:08:25.886 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:08:25.886 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 61 M 2026-02-25T08:08:25.886 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:08:27.408 INFO:teuthology.orchestra.run.trial134.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 3.9 MB/s | 5.1 MB 00:01 2026-02-25T08:08:27.469 INFO:teuthology.orchestra.run.trial134.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 409 kB/s | 25 kB 00:00 2026-02-25T08:08:27.532 INFO:teuthology.orchestra.run.trial134.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 591 kB/s | 37 kB 00:00 2026-02-25T08:08:27.593 INFO:teuthology.orchestra.run.trial134.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 414 kB/s | 26 kB 00:00 2026-02-25T08:08:27.815 INFO:teuthology.orchestra.run.trial134.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 3.8 MB/s | 860 kB 00:00 2026-02-25T08:08:27.877 INFO:teuthology.orchestra.run.trial134.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 4.1 MB/s | 263 kB 00:00 2026-02-25T08:08:29.718 INFO:teuthology.orchestra.run.trial134.stdout:(7/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_6 3.1 MB/s | 5.6 MB 00:01 2026-02-25T08:08:29.781 INFO:teuthology.orchestra.run.trial134.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 716 kB/s | 45 kB 00:00 2026-02-25T08:08:29.912 INFO:teuthology.orchestra.run.trial134.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 1.4 MB/s | 185 kB 00:00 2026-02-25T08:08:30.049 INFO:teuthology.orchestra.run.trial134.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 1.5 MB/s | 204 kB 00:00 2026-02-25T08:08:30.328 INFO:teuthology.orchestra.run.trial134.stdout:(11/37): python3-rados-20.3.0-5338.g6aeb6975.el 1.1 MB/s | 324 kB 00:00 2026-02-25T08:08:30.607 INFO:teuthology.orchestra.run.trial134.stdout:(12/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 1.1 MB/s | 306 kB 00:00 2026-02-25T08:08:30.739 INFO:teuthology.orchestra.run.trial134.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 760 kB/s | 100 kB 00:00 2026-02-25T08:08:31.279 INFO:teuthology.orchestra.run.trial134.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 61 kB/s | 33 kB 00:00 2026-02-25T08:08:32.220 INFO:teuthology.orchestra.run.trial134.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2026-02-25T08:08:32.866 INFO:teuthology.orchestra.run.trial134.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 144 kB/s | 93 kB 00:00 2026-02-25T08:08:32.991 INFO:teuthology.orchestra.run.trial134.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 836 kB/s | 104 kB 00:00 2026-02-25T08:08:33.047 INFO:teuthology.orchestra.run.trial134.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 2.9 MB/s | 164 kB 00:00 2026-02-25T08:08:33.082 INFO:teuthology.orchestra.run.trial134.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.4 MB/s | 160 kB 00:00 2026-02-25T08:08:33.109 INFO:teuthology.orchestra.run.trial134.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.6 MB/s | 45 kB 00:00 2026-02-25T08:08:33.178 INFO:teuthology.orchestra.run.trial134.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 9.4 MB/s | 662 kB 00:00 2026-02-25T08:08:33.209 INFO:teuthology.orchestra.run.trial134.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 9.2 MB/s | 292 kB 00:00 2026-02-25T08:08:33.238 INFO:teuthology.orchestra.run.trial134.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 8.0 MB/s | 240 kB 00:00 2026-02-25T08:08:33.264 INFO:teuthology.orchestra.run.trial134.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.4 MB/s | 37 kB 00:00 2026-02-25T08:08:33.289 INFO:teuthology.orchestra.run.trial134.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.5 MB/s | 66 kB 00:00 2026-02-25T08:08:33.596 INFO:teuthology.orchestra.run.trial134.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.0 MB/s | 308 kB 00:00 2026-02-25T08:08:33.797 INFO:teuthology.orchestra.run.trial134.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2026-02-25T08:08:33.835 INFO:teuthology.orchestra.run.trial134.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 655 kB/s | 25 kB 00:00 2026-02-25T08:08:33.873 INFO:teuthology.orchestra.run.trial134.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.3 MB/s | 49 kB 00:00 2026-02-25T08:08:33.911 INFO:teuthology.orchestra.run.trial134.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2026-02-25T08:08:33.954 INFO:teuthology.orchestra.run.trial134.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 19 MB/s | 838 kB 00:00 2026-02-25T08:08:33.993 INFO:teuthology.orchestra.run.trial134.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 4.8 MB/s | 191 kB 00:00 2026-02-25T08:08:34.041 INFO:teuthology.orchestra.run.trial134.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 33 MB/s | 1.6 MB 00:00 2026-02-25T08:08:39.489 INFO:teuthology.orchestra.run.trial134.stdout:(34/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 617 kB/s | 3.3 MB 00:05 2026-02-25T08:08:43.634 INFO:teuthology.orchestra.run.trial134.stdout:(35/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9 652 kB/s | 11 MB 00:17 2026-02-25T08:08:44.378 INFO:teuthology.orchestra.run.trial134.stdout:(36/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 563 kB/s | 2.7 MB 00:04 2026-02-25T08:09:07.851 INFO:teuthology.orchestra.run.trial134.stdout:(37/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 510 kB/s | 21 MB 00:41 2026-02-25T08:09:07.852 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:09:07.853 INFO:teuthology.orchestra.run.trial134.stdout:Total 1.5 MB/s | 61 MB 00:41 2026-02-25T08:09:07.943 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:09:07.962 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:09:07.962 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:09:08.207 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:09:08.208 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:09:08.498 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:09:08.640 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:09:08.644 INFO:teuthology.orchestra.run.trial134.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:08.692 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:08.693 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:08.705 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:08.708 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:09:08.708 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:08.734 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:08.737 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:09:08.746 INFO:teuthology.orchestra.run.trial134.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:09:08.748 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:09:08.752 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:09:08.756 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:09:08.767 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:09:08.781 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:09:08.782 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:09:08.802 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:09:08.806 INFO:teuthology.orchestra.run.trial134.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:09:08.836 INFO:teuthology.orchestra.run.trial134.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:09:08.840 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:09:08.846 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:09:08.849 INFO:teuthology.orchestra.run.trial134.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:09:08.853 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:09:08.968 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:09:08.968 INFO:teuthology.orchestra.run.trial134.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:09:08.994 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:09:08.996 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:09:09.002 INFO:teuthology.orchestra.run.trial134.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:09:09.251 INFO:teuthology.orchestra.run.trial134.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:09:09.252 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:09:09.268 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:09:09.271 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:09:09.276 INFO:teuthology.orchestra.run.trial134.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:09:09.278 INFO:teuthology.orchestra.run.trial134.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:09:09.279 INFO:teuthology.orchestra.run.trial134.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:09:09.297 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:09:09.414 INFO:teuthology.orchestra.run.trial134.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:09:09.416 INFO:teuthology.orchestra.run.trial134.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:09:09.702 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:09:09.703 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:09:09.703 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:09.747 INFO:teuthology.orchestra.run.trial134.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:09:09.749 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:09:10.689 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:10.691 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:10.952 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:10.955 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:09:10.992 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:09:10.992 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:09:10.992 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:09:10.992 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:10.995 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:15.497 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:15.498 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /sys 2026-02-25T08:09:15.498 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /proc 2026-02-25T08:09:15.498 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /mnt 2026-02-25T08:09:15.498 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /var/tmp 2026-02-25T08:09:15.499 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /home 2026-02-25T08:09:15.499 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /root 2026-02-25T08:09:15.499 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /tmp 2026-02-25T08:09:15.499 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:15.989 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:09:16.012 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:09:16.012 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:09:16.012 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:09:16.012 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:09:16.012 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:09:16.012 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:16.014 INFO:teuthology.orchestra.run.trial134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:09:16.026 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:09:16.026 INFO:teuthology.orchestra.run.trial134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:16.080 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:16.080 INFO:teuthology.orchestra.run.trial134.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:09:16.080 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:16.370 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:09:16.370 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:16.370 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:16.370 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:09:16.371 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:09:16.372 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:09:16.438 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout:Upgraded: 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.439 INFO:teuthology.orchestra.run.trial134.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:09:16.440 INFO:teuthology.orchestra.run.trial134.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.441 INFO:teuthology.orchestra.run.trial134.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:16.442 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:09:16.482 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-test 2026-02-25T08:09:16.807 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:52 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:09:16.891 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:09:16.892 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:09:16.893 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repository Size 2026-02-25T08:09:16.893 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:09:16.893 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:09:16.893 INFO:teuthology.orchestra.run.trial134.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:09:16.893 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-25T08:09:16.893 INFO:teuthology.orchestra.run.trial134.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:09:16.894 INFO:teuthology.orchestra.run.trial134.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:09:16.894 INFO:teuthology.orchestra.run.trial134.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:09:16.894 INFO:teuthology.orchestra.run.trial134.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:09:16.894 INFO:teuthology.orchestra.run.trial134.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:09:16.894 INFO:teuthology.orchestra.run.trial134.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:09:16.894 INFO:teuthology.orchestra.run.trial134.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:09:16.894 INFO:teuthology.orchestra.run.trial134.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:09:16.895 INFO:teuthology.orchestra.run.trial134.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:09:16.895 INFO:teuthology.orchestra.run.trial134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:09:16.895 INFO:teuthology.orchestra.run.trial134.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:09:16.895 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:16.895 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:09:16.895 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:09:16.895 INFO:teuthology.orchestra.run.trial134.stdout:Install 12 Packages 2026-02-25T08:09:16.896 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:09:16.896 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 132 M 2026-02-25T08:09:16.896 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 988 M 2026-02-25T08:09:16.896 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:09:18.101 INFO:teuthology.orchestra.run.trial134.stdout:(1/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 160 kB/s | 153 kB 00:00 2026-02-25T08:09:18.153 INFO:teuthology.orchestra.run.trial134.stdout:(2/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.1 MB/s | 2.1 MB 00:01 2026-02-25T08:09:18.626 INFO:teuthology.orchestra.run.trial134.stdout:(3/12): libatomic-11.5.0-14.el9.x86_64.rpm 43 kB/s | 23 kB 00:00 2026-02-25T08:09:18.665 INFO:teuthology.orchestra.run.trial134.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 13 MB/s | 525 kB 00:00 2026-02-25T08:09:18.672 INFO:teuthology.orchestra.run.trial134.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 26 MB/s | 176 kB 00:00 2026-02-25T08:09:18.680 INFO:teuthology.orchestra.run.trial134.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 27 MB/s | 233 kB 00:00 2026-02-25T08:09:18.713 INFO:teuthology.orchestra.run.trial134.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 31 MB/s | 1.0 MB 00:00 2026-02-25T08:09:18.722 INFO:teuthology.orchestra.run.trial134.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 33 MB/s | 303 kB 00:00 2026-02-25T08:09:18.729 INFO:teuthology.orchestra.run.trial134.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 8.7 MB/s | 64 kB 00:00 2026-02-25T08:09:18.832 INFO:teuthology.orchestra.run.trial134.stdout:(10/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 139 kB/s | 94 kB 00:00 2026-02-25T08:09:19.111 INFO:teuthology.orchestra.run.trial134.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 319 kB/s | 122 kB 00:00 2026-02-25T08:15:02.895 INFO:teuthology.orchestra.run.trial134.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 376 kB/s | 127 MB 05:45 2026-02-25T08:15:02.897 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:15:02.897 INFO:teuthology.orchestra.run.trial134.stdout:Total 390 kB/s | 132 MB 05:46 2026-02-25T08:15:02.968 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:15:02.979 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:15:02.979 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:15:03.411 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:15:03.411 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:15:03.875 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:15:03.884 INFO:teuthology.orchestra.run.trial134.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:15:03.893 INFO:teuthology.orchestra.run.trial134.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:15:03.899 INFO:teuthology.orchestra.run.trial134.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:15:03.902 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:15:03.905 INFO:teuthology.orchestra.run.trial134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:15:03.909 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:15:03.910 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:15:03.928 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:15:03.930 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:15:03.941 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:15:03.949 INFO:teuthology.orchestra.run.trial134.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:15:03.949 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:15:10.914 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:15:10.933 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:15:11.753 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:15:11.753 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:15:11.753 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:15:11.753 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:15:11.753 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:15:11.753 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:15:11.754 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:15:11.754 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:15:11.754 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:15:11.754 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:15:11.754 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:15:11.754 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:15:11.812 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:15:11.812 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:15:11.812 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:15:11.812 INFO:teuthology.orchestra.run.trial134.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:15:11.812 INFO:teuthology.orchestra.run.trial134.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:15:11.813 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:15:11.814 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:15:11.854 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph 2026-02-25T08:15:12.165 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:48 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:15:12.252 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:15:12.255 INFO:teuthology.orchestra.run.trial134.stdout:======================================================================================= 2026-02-25T08:15:12.255 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repository Size 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout:======================================================================================= 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:15:12.256 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:15:12.257 INFO:teuthology.orchestra.run.trial134.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:15:12.258 INFO:teuthology.orchestra.run.trial134.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:15:12.259 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:15:12.260 INFO:teuthology.orchestra.run.trial134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:15:12.261 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:15:12.262 INFO:teuthology.orchestra.run.trial134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout:Installing weak dependencies: 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:15:12.263 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout:======================================================================================= 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout:Install 96 Packages 2026-02-25T08:15:12.264 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:15:12.265 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 226 M 2026-02-25T08:15:12.265 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 1.5 G 2026-02-25T08:15:12.265 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:15:13.082 INFO:teuthology.orchestra.run.trial134.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-25T08:15:14.491 INFO:teuthology.orchestra.run.trial134.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 553 kB/s | 937 kB 00:01 2026-02-25T08:15:15.005 INFO:teuthology.orchestra.run.trial134.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 318 kB/s | 163 kB 00:00 2026-02-25T08:15:17.823 INFO:teuthology.orchestra.run.trial134.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 412 kB/s | 2.0 MB 00:05 2026-02-25T08:15:32.564 INFO:teuthology.orchestra.run.trial134.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 256 kB/s | 4.9 MB 00:19 2026-02-25T08:15:33.278 INFO:teuthology.orchestra.run.trial134.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 80 kB/s | 57 kB 00:00 2026-02-25T08:15:34.325 INFO:teuthology.orchestra.run.trial134.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 176 kB/s | 184 kB 00:01 2026-02-25T08:15:55.422 INFO:teuthology.orchestra.run.trial134.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 374 kB/s | 15 MB 00:40 2026-02-25T08:16:16.479 INFO:teuthology.orchestra.run.trial134.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 360 kB/s | 7.4 MB 00:21 2026-02-25T08:16:16.619 INFO:teuthology.orchestra.run.trial134.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 158 kB/s | 22 kB 00:00 2026-02-25T08:16:17.883 INFO:teuthology.orchestra.run.trial134.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 235 kB/s | 297 kB 00:01 2026-02-25T08:16:18.330 INFO:teuthology.orchestra.run.trial134.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 113 kB/s | 50 kB 00:00 2026-02-25T08:16:18.470 INFO:teuthology.orchestra.run.trial134.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 124 kB/s | 17 kB 00:00 2026-02-25T08:16:20.004 INFO:teuthology.orchestra.run.trial134.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 197 kB/s | 301 kB 00:01 2026-02-25T08:16:23.913 INFO:teuthology.orchestra.run.trial134.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 266 kB/s | 1.0 MB 00:03 2026-02-25T08:16:24.055 INFO:teuthology.orchestra.run.trial134.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.4 MB/s | 351 kB 00:00 2026-02-25T08:16:24.075 INFO:teuthology.orchestra.run.trial134.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 2.0 MB/s | 40 kB 00:00 2026-02-25T08:16:24.096 INFO:teuthology.orchestra.run.trial134.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.3 MB/s | 72 kB 00:00 2026-02-25T08:16:24.147 INFO:teuthology.orchestra.run.trial134.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 15 MB/s | 794 kB 00:00 2026-02-25T08:16:24.170 INFO:teuthology.orchestra.run.trial134.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.9 MB/s | 184 kB 00:00 2026-02-25T08:16:24.193 INFO:teuthology.orchestra.run.trial134.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 11 MB/s | 253 kB 00:00 2026-02-25T08:16:24.237 INFO:teuthology.orchestra.run.trial134.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 28 MB/s | 1.2 MB 00:00 2026-02-25T08:16:24.259 INFO:teuthology.orchestra.run.trial134.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.8 MB/s | 106 kB 00:00 2026-02-25T08:16:24.281 INFO:teuthology.orchestra.run.trial134.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 6.1 MB/s | 135 kB 00:00 2026-02-25T08:16:24.303 INFO:teuthology.orchestra.run.trial134.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.7 MB/s | 150 kB 00:00 2026-02-25T08:16:24.325 INFO:teuthology.orchestra.run.trial134.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.6 MB/s | 126 kB 00:00 2026-02-25T08:16:24.347 INFO:teuthology.orchestra.run.trial134.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 9.6 MB/s | 218 kB 00:00 2026-02-25T08:16:24.370 INFO:teuthology.orchestra.run.trial134.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.9 MB/s | 182 kB 00:00 2026-02-25T08:16:24.392 INFO:teuthology.orchestra.run.trial134.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 12 MB/s | 266 kB 00:00 2026-02-25T08:16:24.651 INFO:teuthology.orchestra.run.trial134.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 114 kB/s | 30 kB 00:00 2026-02-25T08:16:25.243 INFO:teuthology.orchestra.run.trial134.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 5.1 MB/s | 3.0 MB 00:00 2026-02-25T08:16:25.321 INFO:teuthology.orchestra.run.trial134.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 191 kB/s | 15 kB 00:00 2026-02-25T08:16:25.402 INFO:teuthology.orchestra.run.trial134.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 3.0 MB/s | 246 kB 00:00 2026-02-25T08:16:25.482 INFO:teuthology.orchestra.run.trial134.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 2.3 MB/s | 188 kB 00:00 2026-02-25T08:16:25.560 INFO:teuthology.orchestra.run.trial134.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 540 kB/s | 42 kB 00:00 2026-02-25T08:16:25.765 INFO:teuthology.orchestra.run.trial134.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 26 MB/s | 5.3 MB 00:00 2026-02-25T08:16:25.950 INFO:teuthology.orchestra.run.trial134.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 6.0 MB 00:00 2026-02-25T08:16:26.031 INFO:teuthology.orchestra.run.trial134.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 3.0 MB/s | 244 kB 00:00 2026-02-25T08:16:26.112 INFO:teuthology.orchestra.run.trial134.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 3.0 MB/s | 249 kB 00:00 2026-02-25T08:16:26.191 INFO:teuthology.orchestra.run.trial134.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 604 kB/s | 48 kB 00:00 2026-02-25T08:16:26.270 INFO:teuthology.orchestra.run.trial134.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 2.2 MB/s | 177 kB 00:00 2026-02-25T08:16:26.358 INFO:teuthology.orchestra.run.trial134.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2026-02-25T08:16:26.436 INFO:teuthology.orchestra.run.trial134.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 443 kB/s | 35 kB 00:00 2026-02-25T08:16:26.622 INFO:teuthology.orchestra.run.trial134.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 33 MB/s | 6.1 MB 00:00 2026-02-25T08:16:26.704 INFO:teuthology.orchestra.run.trial134.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.3 MB/s | 442 kB 00:00 2026-02-25T08:16:26.783 INFO:teuthology.orchestra.run.trial134.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 978 kB/s | 77 kB 00:00 2026-02-25T08:16:26.863 INFO:teuthology.orchestra.run.trial134.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 3.3 MB/s | 267 kB 00:00 2026-02-25T08:16:26.944 INFO:teuthology.orchestra.run.trial134.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.9 MB/s | 157 kB 00:00 2026-02-25T08:16:27.024 INFO:teuthology.orchestra.run.trial134.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 3.4 MB/s | 277 kB 00:00 2026-02-25T08:16:27.103 INFO:teuthology.orchestra.run.trial134.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 679 kB/s | 54 kB 00:00 2026-02-25T08:16:27.717 INFO:teuthology.orchestra.run.trial134.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 31 MB/s | 19 MB 00:00 2026-02-25T08:16:27.796 INFO:teuthology.orchestra.run.trial134.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 528 kB/s | 42 kB 00:00 2026-02-25T08:16:27.876 INFO:teuthology.orchestra.run.trial134.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.3 MB/s | 189 kB 00:00 2026-02-25T08:16:27.956 INFO:teuthology.orchestra.run.trial134.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-25T08:16:28.538 INFO:teuthology.orchestra.run.trial134.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2026-02-25T08:16:29.765 INFO:teuthology.orchestra.run.trial134.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 702 kB/s | 862 kB 00:01 2026-02-25T08:16:29.810 INFO:teuthology.orchestra.run.trial134.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 12 MB/s | 551 kB 00:00 2026-02-25T08:16:29.814 INFO:teuthology.orchestra.run.trial134.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 4.5 MB/s | 19 kB 00:00 2026-02-25T08:16:29.819 INFO:teuthology.orchestra.run.trial134.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 31 MB/s | 151 kB 00:00 2026-02-25T08:16:29.828 INFO:teuthology.orchestra.run.trial134.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 66 MB/s | 548 kB 00:00 2026-02-25T08:16:29.832 INFO:teuthology.orchestra.run.trial134.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 7.0 MB/s | 29 kB 00:00 2026-02-25T08:16:29.836 INFO:teuthology.orchestra.run.trial134.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 14 MB/s | 60 kB 00:00 2026-02-25T08:16:29.841 INFO:teuthology.orchestra.run.trial134.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 10 MB/s | 43 kB 00:00 2026-02-25T08:16:29.845 INFO:teuthology.orchestra.run.trial134.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 7.6 MB/s | 32 kB 00:00 2026-02-25T08:16:29.849 INFO:teuthology.orchestra.run.trial134.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 3.4 MB/s | 14 kB 00:00 2026-02-25T08:16:29.854 INFO:teuthology.orchestra.run.trial134.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 35 MB/s | 173 kB 00:00 2026-02-25T08:16:29.860 INFO:teuthology.orchestra.run.trial134.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 65 MB/s | 358 kB 00:00 2026-02-25T08:16:29.865 INFO:teuthology.orchestra.run.trial134.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 48 MB/s | 254 kB 00:00 2026-02-25T08:16:29.878 INFO:teuthology.orchestra.run.trial134.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 159 MB/s | 2.0 MB 00:00 2026-02-25T08:16:29.883 INFO:teuthology.orchestra.run.trial134.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 30 MB/s | 144 kB 00:00 2026-02-25T08:16:29.887 INFO:teuthology.orchestra.run.trial134.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 30 MB/s | 139 kB 00:00 2026-02-25T08:16:29.892 INFO:teuthology.orchestra.run.trial134.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 13 MB/s | 56 kB 00:00 2026-02-25T08:16:29.896 INFO:teuthology.orchestra.run.trial134.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 2.6 MB/s | 11 kB 00:00 2026-02-25T08:16:29.900 INFO:teuthology.orchestra.run.trial134.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 4.3 MB/s | 18 kB 00:00 2026-02-25T08:16:29.904 INFO:teuthology.orchestra.run.trial134.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 5.6 MB/s | 23 kB 00:00 2026-02-25T08:16:29.908 INFO:teuthology.orchestra.run.trial134.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 4.8 MB/s | 20 kB 00:00 2026-02-25T08:16:29.912 INFO:teuthology.orchestra.run.trial134.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 4.8 MB/s | 19 kB 00:00 2026-02-25T08:16:29.917 INFO:teuthology.orchestra.run.trial134.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 6.3 MB/s | 26 kB 00:00 2026-02-25T08:16:29.925 INFO:teuthology.orchestra.run.trial134.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 119 MB/s | 1.0 MB 00:00 2026-02-25T08:16:29.930 INFO:teuthology.orchestra.run.trial134.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 18 MB/s | 79 kB 00:00 2026-02-25T08:16:29.935 INFO:teuthology.orchestra.run.trial134.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 19 MB/s | 86 kB 00:00 2026-02-25T08:16:29.939 INFO:teuthology.orchestra.run.trial134.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 14 MB/s | 58 kB 00:00 2026-02-25T08:16:29.943 INFO:teuthology.orchestra.run.trial134.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 4.0 MB/s | 16 kB 00:00 2026-02-25T08:16:29.947 INFO:teuthology.orchestra.run.trial134.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 20 MB/s | 90 kB 00:00 2026-02-25T08:16:29.952 INFO:teuthology.orchestra.run.trial134.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 7.4 MB/s | 31 kB 00:00 2026-02-25T08:16:29.957 INFO:teuthology.orchestra.run.trial134.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 37 MB/s | 188 kB 00:00 2026-02-25T08:16:29.961 INFO:teuthology.orchestra.run.trial134.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 14 MB/s | 59 kB 00:00 2026-02-25T08:16:29.966 INFO:teuthology.orchestra.run.trial134.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 27 MB/s | 125 kB 00:00 2026-02-25T08:16:29.970 INFO:teuthology.orchestra.run.trial134.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 8.5 MB/s | 36 kB 00:00 2026-02-25T08:16:29.974 INFO:teuthology.orchestra.run.trial134.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 20 MB/s | 86 kB 00:00 2026-02-25T08:16:29.979 INFO:teuthology.orchestra.run.trial134.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 20 MB/s | 90 kB 00:00 2026-02-25T08:16:29.983 INFO:teuthology.orchestra.run.trial134.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 11 MB/s | 48 kB 00:00 2026-02-25T08:16:29.987 INFO:teuthology.orchestra.run.trial134.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 5.5 MB/s | 22 kB 00:00 2026-02-25T08:16:29.991 INFO:teuthology.orchestra.run.trial134.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 4.9 MB/s | 20 kB 00:00 2026-02-25T08:19:25.807 INFO:teuthology.orchestra.run.trial134.stdout:(95/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 362 kB/s | 88 MB 04:07 2026-02-25T08:19:33.402 INFO:teuthology.orchestra.run.trial134.stdout:(96/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 217 kB/s | 51 MB 03:59 2026-02-25T08:19:33.405 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:33.406 INFO:teuthology.orchestra.run.trial134.stdout:Total 885 kB/s | 226 MB 04:21 2026-02-25T08:19:33.771 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:19:33.808 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:19:33.808 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:19:34.615 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:19:34.615 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:19:35.447 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:19:35.449 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:19:35.453 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:19:35.474 INFO:teuthology.orchestra.run.trial134.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:19:35.508 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:19:35.513 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:19:35.516 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:19:35.520 INFO:teuthology.orchestra.run.trial134.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:19:35.543 INFO:teuthology.orchestra.run.trial134.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:19:35.556 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:19:35.559 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:19:35.565 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:19:35.567 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:19:35.578 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:19:35.584 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:19:35.609 INFO:teuthology.orchestra.run.trial134.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:19:35.612 INFO:teuthology.orchestra.run.trial134.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:19:35.662 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:19:35.674 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:19:35.677 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:19:35.684 INFO:teuthology.orchestra.run.trial134.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:19:35.689 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:19:35.693 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:19:35.697 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:19:35.711 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:19:35.716 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:19:35.721 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:19:35.732 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:19:35.741 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:19:35.747 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:19:35.767 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:19:35.779 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:19:35.794 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:19:35.799 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:19:35.807 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:19:35.847 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:19:36.171 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:19:36.181 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:19:36.185 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:19:36.191 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:19:36.192 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:19:36.220 INFO:teuthology.orchestra.run.trial134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:19:36.264 INFO:teuthology.orchestra.run.trial134.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:19:36.272 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:19:36.278 INFO:teuthology.orchestra.run.trial134.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:19:36.281 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:19:36.287 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:19:36.291 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:19:36.298 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:19:36.301 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:19:36.329 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:19:36.339 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:19:36.345 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:19:36.351 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:19:36.386 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:19:36.609 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:19:36.632 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:19:36.636 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:19:36.695 INFO:teuthology.orchestra.run.trial134.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:19:36.697 INFO:teuthology.orchestra.run.trial134.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:19:36.718 INFO:teuthology.orchestra.run.trial134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:19:37.043 INFO:teuthology.orchestra.run.trial134.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:19:37.116 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:19:37.785 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:19:37.801 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:19:37.811 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:19:37.818 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:19:37.832 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:19:37.847 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:19:37.922 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:19:37.931 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:19:37.954 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:19:37.983 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:19:38.034 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:19:38.040 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:19:38.043 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:19:38.045 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:19:38.063 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:19:38.063 INFO:teuthology.orchestra.run.trial134.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:19:38.063 INFO:teuthology.orchestra.run.trial134.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:19:38.063 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:38.070 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:19:38.096 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:19:38.096 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:19:38.096 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:38.133 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:19:38.757 INFO:teuthology.orchestra.run.trial134.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:19:44.416 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:19:44.479 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:19:44.481 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:19:44.494 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:19:44.494 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:44.494 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:19:44.494 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:44.504 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:19:44.528 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:19:44.528 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:44.528 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:19:44.528 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:19:44.528 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:19:44.529 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:44.586 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:19:44.587 INFO:teuthology.orchestra.run.trial134.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:19:44.596 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:19:44.620 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:19:44.621 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:19:47.634 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:19:47.640 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:19:48.119 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:19:48.120 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:19:48.132 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:19:48.133 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:19:48.194 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:19:48.235 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:19:48.236 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:19:48.256 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:19:48.257 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:48.257 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:19:48.257 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:19:48.257 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:19:48.257 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:48.270 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:19:48.494 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:19:48.495 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:19:48.516 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:19:48.516 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:48.517 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:19:48.517 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:19:48.517 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:19:48.517 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:48.602 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:19:48.621 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:19:48.622 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:48.622 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:19:48.622 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:19:48.622 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:19:48.622 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:48.623 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:19:48.629 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:19:48.652 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:19:50.016 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:19:50.016 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:19:50.016 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:19:50.016 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:19:50.016 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:19:50.016 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:19:50.017 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:19:50.018 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:19:50.019 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:19:50.020 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:19:50.021 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:19:50.022 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:19:50.022 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:19:50.022 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:19:50.022 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:19:50.022 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:19:50.087 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:19:50.087 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:50.087 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:19:50.087 INFO:teuthology.orchestra.run.trial134.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.088 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:19:50.089 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:19:50.090 INFO:teuthology.orchestra.run.trial134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:19:50.091 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:19:50.092 INFO:teuthology.orchestra.run.trial134.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:19:50.093 INFO:teuthology.orchestra.run.trial134.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:19:50.094 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:50.095 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:50.155 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-base 2026-02-25T08:19:50.473 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:26 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:50.538 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:50.557 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:50.558 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:50.558 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:50.579 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install cephadm 2026-02-25T08:19:50.903 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:26 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:50.968 INFO:teuthology.orchestra.run.trial134.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:50.987 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:50.988 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:50.988 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:51.009 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:19:51.318 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:27 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:51.402 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:51.402 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:51.402 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 149 k 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 455 k 2026-02-25T08:19:51.403 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:19:52.405 INFO:teuthology.orchestra.run.trial134.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 149 kB/s | 149 kB 00:01 2026-02-25T08:19:52.406 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:52.407 INFO:teuthology.orchestra.run.trial134.stdout:Total 149 kB/s | 149 kB 00:01 2026-02-25T08:19:52.407 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:19:52.411 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:19:52.411 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:19:52.442 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:19:52.442 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:19:52.499 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:19:52.501 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:19:52.524 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:19:52.524 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:52.524 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:19:52.524 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:52.824 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:19:52.825 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:52.825 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:19:52.825 INFO:teuthology.orchestra.run.trial134.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:52.825 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:52.825 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:52.850 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr 2026-02-25T08:19:53.168 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:29 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:53.233 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:53.252 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:53.252 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:53.252 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:53.274 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:19:53.588 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:29 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:53.653 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:53.672 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:53.672 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:53.673 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:53.693 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:19:54.005 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:30 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:54.070 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:54.089 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:54.090 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:54.090 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:54.111 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:19:54.425 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:30 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:54.489 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:54.508 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:54.509 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:54.509 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:54.530 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:19:54.843 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:30 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:54.908 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:54.927 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:54.927 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:54.927 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:54.948 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd 2026-02-25T08:19:55.269 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:31 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:55.335 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:55.354 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:55.355 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:55.355 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:55.377 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-classic 2026-02-25T08:19:55.694 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:31 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:55.759 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:55.778 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:55.779 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:55.779 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:55.800 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-fuse 2026-02-25T08:19:56.115 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:32 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:56.199 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:56.200 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:56.200 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-25T08:19:56.200 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:56.200 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 914 k 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 2.9 M 2026-02-25T08:19:56.201 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:19:58.475 INFO:teuthology.orchestra.run.trial134.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 402 kB/s | 914 kB 00:02 2026-02-25T08:19:58.475 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:58.475 INFO:teuthology.orchestra.run.trial134.stdout:Total 402 kB/s | 914 kB 00:02 2026-02-25T08:19:58.475 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:19:58.480 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:19:58.480 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:19:58.514 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:19:58.514 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:19:58.604 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:19:58.622 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:58.874 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:58.935 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:58.935 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:58.935 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:19:58.936 INFO:teuthology.orchestra.run.trial134.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:58.936 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:58.936 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:58.963 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-volume 2026-02-25T08:19:59.283 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:35 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:59.348 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:59.366 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:59.367 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:19:59.367 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:19:59.389 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados-devel 2026-02-25T08:19:59.703 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:35 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:19:59.787 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:19:59.788 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:59.788 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-25T08:19:59.788 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:59.788 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:19:59.788 INFO:teuthology.orchestra.run.trial134.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:19:59.788 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:59.789 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:19:59.789 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:19:59.789 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-25T08:19:59.789 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:19:59.789 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 124 k 2026-02-25T08:19:59.789 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 434 k 2026-02-25T08:19:59.789 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:20:00.353 INFO:teuthology.orchestra.run.trial134.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 219 kB/s | 124 kB 00:00 2026-02-25T08:20:00.353 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:20:00.353 INFO:teuthology.orchestra.run.trial134.stdout:Total 219 kB/s | 124 kB 00:00 2026-02-25T08:20:00.353 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:20:00.356 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:20:00.356 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:20:00.388 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:20:00.388 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:20:00.442 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:20:00.460 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:00.548 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:00.610 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:00.610 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:00.610 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:20:00.610 INFO:teuthology.orchestra.run.trial134.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:20:00.610 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:00.610 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:00.640 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs2 2026-02-25T08:20:00.957 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:36 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:01.022 INFO:teuthology.orchestra.run.trial134.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:01.041 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:01.041 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:01.041 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:01.063 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs-devel 2026-02-25T08:20:01.374 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:37 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:01.458 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:01.458 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 37 k 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 171 k 2026-02-25T08:20:01.459 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:20:01.816 INFO:teuthology.orchestra.run.trial134.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 104 kB/s | 37 kB 00:00 2026-02-25T08:20:01.817 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:20:01.817 INFO:teuthology.orchestra.run.trial134.stdout:Total 104 kB/s | 37 kB 00:00 2026-02-25T08:20:01.817 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:20:01.820 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:20:01.820 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:20:01.822 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:20:01.823 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:20:01.845 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:20:01.862 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:01.923 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:01.985 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:01.985 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:01.985 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:20:01.985 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:20:01.985 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:01.985 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:02.012 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados2 2026-02-25T08:20:02.330 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:38 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:02.395 INFO:teuthology.orchestra.run.trial134.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:02.412 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:02.413 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:02.413 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:02.435 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librbd1 2026-02-25T08:20:02.749 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:38 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:02.814 INFO:teuthology.orchestra.run.trial134.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:02.831 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:02.832 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:02.832 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:02.853 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rados 2026-02-25T08:20:03.170 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:39 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:03.235 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:03.253 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:03.253 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:03.254 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:03.276 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rgw 2026-02-25T08:20:03.590 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:39 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:03.655 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:03.672 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:03.673 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:03.673 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:03.695 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-cephfs 2026-02-25T08:20:04.007 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:40 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:04.072 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:04.089 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:04.090 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:04.090 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:04.112 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rbd 2026-02-25T08:20:04.426 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:40 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:04.492 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:04.509 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:04.510 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:04.510 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:04.531 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-fuse 2026-02-25T08:20:04.847 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:40 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:04.933 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout: Package Architecture Version Repository Size 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:04.934 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-25T08:20:04.935 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:04.935 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 86 k 2026-02-25T08:20:04.935 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 236 k 2026-02-25T08:20:04.935 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:20:06.122 INFO:teuthology.orchestra.run.trial134.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 73 kB/s | 86 kB 00:01 2026-02-25T08:20:06.122 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:20:06.122 INFO:teuthology.orchestra.run.trial134.stdout:Total 73 kB/s | 86 kB 00:01 2026-02-25T08:20:06.123 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:20:06.127 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:20:06.127 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:20:06.158 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:20:06.158 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:20:06.210 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:20:06.227 INFO:teuthology.orchestra.run.trial134.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:06.312 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:06.376 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:06.376 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:06.376 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:20:06.377 INFO:teuthology.orchestra.run.trial134.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:20:06.377 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:06.377 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:06.406 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-mirror 2026-02-25T08:20:06.724 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:42 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:06.808 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:06.808 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:06.809 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 2.6 M 2026-02-25T08:20:06.810 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 11 M 2026-02-25T08:20:06.810 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:20:18.273 INFO:teuthology.orchestra.run.trial134.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 232 kB/s | 2.6 MB 00:11 2026-02-25T08:20:18.274 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:20:18.274 INFO:teuthology.orchestra.run.trial134.stdout:Total 232 kB/s | 2.6 MB 00:11 2026-02-25T08:20:18.274 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:20:18.279 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:20:18.280 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:20:18.318 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:20:18.318 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:20:18.485 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:20:18.486 INFO:teuthology.orchestra.run.trial134.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:18.517 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:18.518 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:20:18.518 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:20:18.518 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:20:18.518 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:20:18.518 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:18.847 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:18.847 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:18.847 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:20:18.847 INFO:teuthology.orchestra.run.trial134.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:20:18.847 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:18.847 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:18.875 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-nbd 2026-02-25T08:20:19.197 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:55 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:19.280 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:19.281 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:19.281 INFO:teuthology.orchestra.run.trial134.stdout: Package Architecture Version Repository Size 2026-02-25T08:20:19.281 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:19.281 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:20:19.281 INFO:teuthology.orchestra.run.trial134.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 166 k 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 496 k 2026-02-25T08:20:19.282 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:20:19.872 INFO:teuthology.orchestra.run.trial134.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 282 kB/s | 166 kB 00:00 2026-02-25T08:20:19.873 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:20:19.873 INFO:teuthology.orchestra.run.trial134.stdout:Total 281 kB/s | 166 kB 00:00 2026-02-25T08:20:19.873 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:20:19.878 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:20:19.878 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:20:19.909 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:20:19.909 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:20:19.965 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:20:19.983 INFO:teuthology.orchestra.run.trial134.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:20.074 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:20.137 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:20:20.137 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:20.138 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:20:20.138 INFO:teuthology.orchestra.run.trial134.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:20:20.138 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:20.138 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:20.167 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:20:20.486 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:56 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:20.551 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:20:20.568 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:20.569 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:20:20.569 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:20:20.591 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:20:20.913 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:11:56 ago on Wed 25 Feb 2026 08:08:24 AM UTC. 2026-02-25T08:20:20.996 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:20:20.997 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:20.997 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-25T08:20:20.997 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:20.997 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout:Installing weak dependencies: 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout:Install 2 Packages 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 445 M 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 2.7 G 2026-02-25T08:20:20.998 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-25T08:20:41.558 INFO:teuthology.orchestra.run.trial134.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 600 kB/s | 12 MB 00:20 2026-02-25T08:39:04.537 INFO:teuthology.orchestra.run.trial134.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 394 kB/s | 433 MB 18:43 2026-02-25T08:39:04.538 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:39:04.539 INFO:teuthology.orchestra.run.trial134.stdout:Total 405 kB/s | 445 MB 18:43 2026-02-25T08:39:04.539 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-25T08:39:04.545 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-25T08:39:04.545 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-25T08:39:05.922 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-25T08:39:05.922 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-25T08:39:09.511 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-25T08:39:31.802 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:39:31.822 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:33.724 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:33.724 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:39:33.788 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:33.788 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:39:33.789 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-25T08:39:33.789 INFO:teuthology.orchestra.run.trial134.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:39:33.789 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:39:33.789 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:39:33.789 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:33.864 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:33.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:39:34.028 DEBUG:teuthology.orchestra.run.trial134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:39:34.050 INFO:teuthology.orchestra.run.trial134.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:34.050 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:34.051 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:39:34.053 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T08:39:34.053 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:39:34.053 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:39:34.117 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T08:39:34.117 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:39:34.117 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:39:34.181 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:39:34.246 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T08:39:34.247 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:39:34.247 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:39:34.310 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:39:34.373 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T08:39:34.374 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:39:34.374 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:39:34.436 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:39:34.500 INFO:teuthology.run_tasks:Running task pexec... 2026-02-25T08:39:34.518 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-25T08:39:34.519 DEBUG:teuthology.orchestra.run.trial134:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:39:34.543 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:39:34.543 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:39:34.594 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:34.594 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:39:34.609 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:39:34.609 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:39:34.609 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:39:34.609 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:39:34.618 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:39:34.618 INFO:teuthology.task.install:extra packages: [] 2026-02-25T08:39:34.618 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:39:34.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:39:34.779 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:39:34.779 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:39:35.137 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:39:35.137 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:39:35.137 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:39:35.163 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:39:35.163 DEBUG:teuthology.orchestra.run.trial134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:39:35.228 DEBUG:teuthology.orchestra.run.trial134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:39:35.292 DEBUG:teuthology.orchestra.run.trial134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:39:35.347 INFO:teuthology.orchestra.run.trial134.stdout:check_obsoletes = 1 2026-02-25T08:39:35.357 DEBUG:teuthology.orchestra.run.trial134:> sudo yum clean all 2026-02-25T08:39:35.519 INFO:teuthology.orchestra.run.trial134.stdout:56 files removed 2026-02-25T08:39:35.535 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-radosgw 2026-02-25T08:39:36.409 INFO:teuthology.orchestra.run.trial134.stdout:ceph packages for x86_64 131 kB/s | 94 kB 00:00 2026-02-25T08:39:37.081 INFO:teuthology.orchestra.run.trial134.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-25T08:39:37.657 INFO:teuthology.orchestra.run.trial134.stdout:ceph source packages 3.8 kB/s | 2.1 kB 00:00 2026-02-25T08:39:38.768 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.9 MB 00:01 2026-02-25T08:39:39.747 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - AppStream 60 MB/s | 27 MB 00:00 2026-02-25T08:39:43.047 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 8.0 MB 00:00 2026-02-25T08:39:44.783 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 20 kB 00:00 2026-02-25T08:39:45.739 INFO:teuthology.orchestra.run.trial134.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-25T08:39:48.954 INFO:teuthology.orchestra.run.trial134.stdout:lab-extras 2.9 MB/s | 50 kB 00:00 2026-02-25T08:39:50.128 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:50.147 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:50.147 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:50.147 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:50.172 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-test 2026-02-25T08:39:50.494 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:50.559 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:50.577 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:50.577 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:50.577 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:50.599 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph 2026-02-25T08:39:50.912 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:50.977 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:50.995 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:50.995 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:50.996 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:51.017 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-base 2026-02-25T08:39:51.338 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:51.404 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:51.421 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:51.422 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:51.422 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:51.444 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install cephadm 2026-02-25T08:39:51.766 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:51.831 INFO:teuthology.orchestra.run.trial134.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:51.849 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:51.849 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:51.850 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:51.871 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:39:52.192 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:52.257 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:52.275 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:52.275 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:52.275 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:52.298 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr 2026-02-25T08:39:52.620 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:52.685 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:52.703 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:52.703 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:52.704 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:52.725 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:39:53.043 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:53.108 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:53.126 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:53.127 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:53.127 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:53.149 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:39:53.474 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:53.539 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:53.556 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:53.557 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:53.557 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:53.579 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:39:53.894 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:53.960 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:53.978 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:53.978 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:53.978 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:54.000 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:39:54.322 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:54.388 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:54.405 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:54.406 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:54.406 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:54.428 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd 2026-02-25T08:39:54.745 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:54.813 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:54.830 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:54.831 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:54.831 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:54.853 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-classic 2026-02-25T08:39:55.173 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:55.239 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:55.256 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:55.257 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:55.257 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:55.279 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-fuse 2026-02-25T08:39:55.602 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:55.668 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:55.685 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:55.686 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:55.686 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:55.708 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-volume 2026-02-25T08:39:56.026 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:56.091 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:39:56.109 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:56.110 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:56.110 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:56.131 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados-devel 2026-02-25T08:39:56.451 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:56.517 INFO:teuthology.orchestra.run.trial134.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:56.535 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:56.535 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:56.535 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:56.557 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs2 2026-02-25T08:39:56.878 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:56.943 INFO:teuthology.orchestra.run.trial134.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:56.961 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:56.961 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:56.961 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:56.983 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs-devel 2026-02-25T08:39:57.303 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:57.369 INFO:teuthology.orchestra.run.trial134.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:57.387 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:57.387 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:57.388 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:57.409 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados2 2026-02-25T08:39:57.734 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:57.799 INFO:teuthology.orchestra.run.trial134.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:57.817 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:57.818 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:57.818 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:57.840 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librbd1 2026-02-25T08:39:58.160 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:58.227 INFO:teuthology.orchestra.run.trial134.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:58.245 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:58.245 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:58.245 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:58.267 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rados 2026-02-25T08:39:58.592 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:58.659 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:58.677 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:58.677 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:58.678 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:58.700 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rgw 2026-02-25T08:39:59.023 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:59.089 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:59.106 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:59.107 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:59.107 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:59.129 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-cephfs 2026-02-25T08:39:59.445 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:59.510 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:59.528 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:59.529 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:59.529 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:59.551 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rbd 2026-02-25T08:39:59.871 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:39:59.938 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:39:59.956 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:39:59.957 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:39:59.957 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:39:59.979 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-fuse 2026-02-25T08:40:00.298 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:40:00.364 INFO:teuthology.orchestra.run.trial134.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:00.381 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:40:00.382 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:40:00.382 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:40:00.405 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-mirror 2026-02-25T08:40:00.727 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:40:00.793 INFO:teuthology.orchestra.run.trial134.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:00.810 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:40:00.811 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:40:00.811 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:40:00.833 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-nbd 2026-02-25T08:40:01.153 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:13 ago on Wed 25 Feb 2026 08:39:48 AM UTC. 2026-02-25T08:40:01.219 INFO:teuthology.orchestra.run.trial134.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:01.237 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-25T08:40:01.237 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-25T08:40:01.237 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-25T08:40:01.264 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:01.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:01.427 DEBUG:teuthology.orchestra.run.trial134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:01.448 INFO:teuthology.orchestra.run.trial134.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:01.449 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:01.449 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:01.452 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-25T08:40:01.452 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T08:40:01.564 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T08:40:01.564 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:01.589 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T08:40:01.589 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:01.653 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T08:40:01.654 DEBUG:teuthology.orchestra.run.trial134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:01.726 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T08:40:01.726 INFO:tasks.ceph:config {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T08:40:01.726 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072', 'branch': 'wip-logstoremanager', 'description': 'crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/segmented$/{crimson_seastore_segmented} rados}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '69072', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}, 'osd': {'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 869, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d', 'targets': {'trial134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJu4mEylc8n2tExh8UuvaV6MfNEUJLoPm8uE+pYer8rpGwDPLHkpyQMHF/LGf6RIm5SZzYyeWSxV1jcdMIBZ8ng='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 30, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300, 'osd pool default crimson': True}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'crimson_compat': True, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cephtool/test_crimson_dump_metrics.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_07:55:35', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T08:40:01.726 DEBUG:teuthology.orchestra.run.trial134:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:01.774 DEBUG:teuthology.orchestra.run.trial134:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:01.837 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:01.837 DEBUG:teuthology.orchestra.run.trial134:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:01.894 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:01.894 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:01.951 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:01.951 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:01.951 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1294 Links: 1 2026-02-25T08:40:01.952 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:01.952 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:01.952 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-25 08:20:18.655425954 +0000 2026-02-25T08:40:01.952 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:01.952 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:01.952 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:01.952 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:02.014 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-25T08:40:02.014 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-25T08:40:02.014 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.000164387 s, 3.1 MB/s 2026-02-25T08:40:02.015 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:02.073 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:02.129 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:02.129 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:02.129 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-25T08:40:02.129 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:02.129 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:02.129 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-25 08:20:18.655425954 +0000 2026-02-25T08:40:02.129 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.130 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.130 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.130 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:02.193 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-25T08:40:02.193 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-25T08:40:02.193 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.000165258 s, 3.1 MB/s 2026-02-25T08:40:02.194 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:02.252 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1289 Links: 1 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-25 08:20:18.656425924 +0000 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.308 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.309 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:02.370 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-25T08:40:02.370 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-25T08:40:02.370 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.000164417 s, 3.1 MB/s 2026-02-25T08:40:02.372 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:02.428 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:02.484 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1287 Links: 1 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-25 08:20:18.656425924 +0000 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.485 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-25 08:09:09.729506299 +0000 2026-02-25T08:40:02.486 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:02.547 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-25T08:40:02.548 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-25T08:40:02.548 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.000151292 s, 3.4 MB/s 2026-02-25T08:40:02.549 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:02.605 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:02.605 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-25T08:40:02.605 INFO:tasks.ceph:Generating config... 2026-02-25T08:40:02.608 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:02.608 INFO:tasks.ceph:[osd] crimson cpu num = 2 2026-02-25T08:40:02.608 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-02-25T08:40:02.608 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-25T08:40:02.608 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-25T08:40:02.609 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-25T08:40:02.610 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-25T08:40:02.611 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-25T08:40:02.611 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T08:40:02.611 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T08:40:02.611 INFO:tasks.ceph:[mon] debug mon = 30 2026-02-25T08:40:02.611 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-25T08:40:02.611 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-25T08:40:02.611 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-25T08:40:02.611 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-25T08:40:02.611 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T08:40:02.612 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:02.686 INFO:teuthology.orchestra.run.trial134.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T08:40:02.689 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T08:40:02.773 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:02.838 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.134'), ('mon.b', '[v2:10.20.193.134:3301,v1:10.20.193.134:6790]'), ('mon.c', '[v2:10.20.193.134:3302,v1:10.20.193.134:6791]')] 2026-02-25T08:40:02.839 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.134,[v2:10.20.193.134:3301,v1:10.20.193.134:6790],[v2:10.20.193.134:3302,v1:10.20.193.134:6791]', 'enable experimental unrecoverable data corrupting features': 'crimson'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'crimson cpu num': 2, 'osd min pg log entries': 5, 'bluestore zero block detection': True, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd objectstore': 'seastore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 30, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-25T08:40:02.840 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:02.840 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:02.895 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.134 --addv b '[v2:10.20.193.134:3301,v1:10.20.193.134:6790]' --addv c '[v2:10.20.193.134:3302,v1:10.20.193.134:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:02.978 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:02.976+0000 7f5e44035f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:02.978 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:02.976+0000 7f5e44035f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:02.978 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:02.976+0000 7f5e44035f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:02.979 INFO:teuthology.orchestra.run.trial134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:monmaptool: generated fsid e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:setting min_mon_release = tentacle 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:epoch 0 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:fsid e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:last_changed 2026-02-25T08:40:02.977795+0000 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:created 2026-02-25T08:40:02.977795+0000 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:min_mon_release 20 (tentacle) 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:election_strategy: 1 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:0: [v2:10.20.193.134:3300/0,v1:10.20.193.134:6789/0] mon.a 2026-02-25T08:40:02.980 INFO:teuthology.orchestra.run.trial134.stdout:1: [v2:10.20.193.134:3301/0,v1:10.20.193.134:6790/0] mon.b 2026-02-25T08:40:02.981 INFO:teuthology.orchestra.run.trial134.stdout:2: [v2:10.20.193.134:3302/0,v1:10.20.193.134:6791/0] mon.c 2026-02-25T08:40:02.981 INFO:teuthology.orchestra.run.trial134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-25T08:40:02.981 DEBUG:teuthology.orchestra.run.trial134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:03.036 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b... 2026-02-25T08:40:03.037 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout:[global] 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout: chdir = "" 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout: auth supported = cephx 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout: filestore xattr use omap = true 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.117 INFO:teuthology.orchestra.run.trial134.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: auth debug = true 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: ms die on old message = true 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: ms die on bug = true 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:03.118 INFO:teuthology.orchestra.run.trial134.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: osd pool default size = 2 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon osd allow pg remap = true 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on too few osds = false 2026-02-25T08:40:03.119 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: osd default data pool replay window = 5 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: mon allow pool delete = true 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: mon cluster log file level = debug 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: debug asserts on shutdown = true 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: mon health detail to clog = false 2026-02-25T08:40:03.120 INFO:teuthology.orchestra.run.trial134.stdout: mon host = "10.20.193.134,[v2:10.20.193.134:3301,v1:10.20.193.134:6790],[v2:10.20.193.134:3302,v1:10.20.193.134:6791]" 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: fsid = e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout:[osd] 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: osd journal size = 100 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: osd scrub max interval = 600 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: osd mclock skip benchmark = True 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.121 INFO:teuthology.orchestra.run.trial134.stdout: osd recover clone overlap = true 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd debug shutdown = true 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd debug op order = true 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd debug trim objects = true 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd open classes on start = true 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd debug pg log writeout = true 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.122 INFO:teuthology.orchestra.run.trial134.stdout: osd map max advance = 10 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: journal zero on create = true 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: bdev debug aio = true 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: osd debug misdirected ops = true 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: crimson cpu num = 2 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: osd min pg log entries = 5 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: bluestore zero block detection = True 2026-02-25T08:40:03.123 INFO:teuthology.orchestra.run.trial134.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug ms = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug osd = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore backref = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore cache = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore cleaner = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore device = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore epm = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore journal = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore lba = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore odata = 20 2026-02-25T08:40:03.124 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore omap = 20 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore onode = 20 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore t = 20 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore tm = 20 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: osd debug verify cached snaps = True 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: osd debug verify missing on start = True 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: osd mclock override recovery settings = True 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: osd objectstore = seastore 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: osd op queue = debug_random 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: osd op queue cut off = debug_random 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.125 INFO:teuthology.orchestra.run.trial134.stdout:[mgr] 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug ms = 1 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug mgr = 20 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug mon = 20 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug auth = 20 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout:[mon] 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug ms = 1 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug mon = 30 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug paxos = 20 2026-02-25T08:40:03.126 INFO:teuthology.orchestra.run.trial134.stdout: debug auth = 20 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: mon data avail warn = 5 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: mon osd prime pg temp = true 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.127 INFO:teuthology.orchestra.run.trial134.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: # 1m isn't quite enough 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: mon scrub interval = 300 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: osd pool default crimson = True 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout:[client] 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: rgw cache enabled = true 2026-02-25T08:40:03.128 INFO:teuthology.orchestra.run.trial134.stdout: rgw enable ops log = true 2026-02-25T08:40:03.129 INFO:teuthology.orchestra.run.trial134.stdout: rgw enable usage log = true 2026-02-25T08:40:03.129 INFO:teuthology.orchestra.run.trial134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:03.129 INFO:teuthology.orchestra.run.trial134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:03.129 INFO:teuthology.orchestra.run.trial134.stdout:[mon.a] 2026-02-25T08:40:03.129 INFO:teuthology.orchestra.run.trial134.stdout:[mon.b] 2026-02-25T08:40:03.129 INFO:teuthology.orchestra.run.trial134.stdout:[mon.c] 2026-02-25T08:40:03.130 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T08:40:03.130 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T08:40:03.221 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T08:40:03.221 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:03.221 DEBUG:teuthology.orchestra.run.trial134:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T08:40:03.237 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:03.238 DEBUG:teuthology.orchestra.run.trial134:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T08:40:03.294 INFO:tasks.ceph:Sending monmap to node ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:40:03.295 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:03.295 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:03.295 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:03.368 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:03.368 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:03.423 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T08:40:03.424 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T08:40:03.424 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:03.513 INFO:teuthology.orchestra.run.trial134.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:03.516 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T08:40:03.516 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T08:40:03.516 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:03.559 INFO:teuthology.orchestra.run.trial134.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:03.570 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T08:40:03.571 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-25T08:40:03.571 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:03.635 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:03.635 INFO:tasks.ceph:role: osd.0 2026-02-25T08:40:03.635 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:40:03.635 DEBUG:teuthology.orchestra.run.trial134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:03.700 INFO:teuthology.orchestra.run.trial134.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:03.701 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:03.701 INFO:teuthology.orchestra.run.trial134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:04.268 INFO:teuthology.orchestra.run.trial134.stdout:Discarding blocks...Done. 2026-02-25T08:40:04.270 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial134.front.sepia.ceph.com -o noatime 2026-02-25T08:40:04.271 DEBUG:teuthology.orchestra.run.trial134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:04.340 DEBUG:teuthology.orchestra.run.trial134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:04.410 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:04.473 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:04.474 INFO:tasks.ceph:role: osd.1 2026-02-25T08:40:04.474 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:40:04.474 DEBUG:teuthology.orchestra.run.trial134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:04.538 INFO:teuthology.orchestra.run.trial134.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:04.539 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:04.539 INFO:teuthology.orchestra.run.trial134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:05.099 INFO:teuthology.orchestra.run.trial134.stdout:Discarding blocks...Done. 2026-02-25T08:40:05.102 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial134.front.sepia.ceph.com -o noatime 2026-02-25T08:40:05.102 DEBUG:teuthology.orchestra.run.trial134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:05.170 DEBUG:teuthology.orchestra.run.trial134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:05.236 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:05.300 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:05.300 INFO:tasks.ceph:role: osd.2 2026-02-25T08:40:05.300 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:40:05.301 DEBUG:teuthology.orchestra.run.trial134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-25T08:40:05.364 INFO:teuthology.orchestra.run.trial134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:05.365 INFO:teuthology.orchestra.run.trial134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:05.921 INFO:teuthology.orchestra.run.trial134.stdout:Discarding blocks...Done. 2026-02-25T08:40:05.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial134.front.sepia.ceph.com -o noatime 2026-02-25T08:40:05.923 DEBUG:teuthology.orchestra.run.trial134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:05.995 DEBUG:teuthology.orchestra.run.trial134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:06.065 DEBUG:teuthology.orchestra.run.trial134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:06.215 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:06.275 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:06,274 seastar - Reactor backend: linux-aio 2026-02-25T08:40:06.292 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:06,291 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:06.292 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:06,291 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:06.294 INFO:teuthology.orchestra.run.trial134.stderr:==39935==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:06.350 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:06,349 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:06.351 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:06,349 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:06.351 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:06,349 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:06.351 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:06,349 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-25T08:40:06.351 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-25 08:40:06,350 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:08.416 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:08.416 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-25T08:40:08.416 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-25T08:40:08.416 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:08.416 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:08.416 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:40:08.436 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:08.437 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:08.478 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:08.478 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:08.485 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:08,484 seastar - Reactor backend: linux-aio 2026-02-25T08:40:08.499 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:08,498 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:08.499 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:08,498 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:08.501 INFO:teuthology.orchestra.run.trial134.stderr:==39933==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:08.501 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:08,501 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:08.559 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-25 08:40:08,558 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:08.640 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12985d /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9f02d0e 0x9f03487 0x9f03972 0x9f03d05 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:08.673 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xe29e44 0x1138071 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba26b2 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:08.716 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x4c96080 0x4c9fba4 0x9ef6807 0x9ef6976 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba26b2 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:08.767 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69801 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:08.810 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a68f88 0x7d56d00 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:08.825 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a68a2b 0x7d56cf3 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:08.868 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc08 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:08.983 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb0570 /lib64/libasan.so.8+0xb1437 0xb7ac87 0x10a27ac 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:09.066 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129747 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:09.123 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:11.312 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:11.312 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-25T08:40:11.312 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-25T08:40:11.312 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:11.313 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:11.313 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:40:11.345 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:11.371 DEBUG:teuthology.orchestra.run.trial134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:11.521 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:11.577 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:11,576 seastar - Reactor backend: linux-aio 2026-02-25T08:40:11.595 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:11,594 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:11.595 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:11,594 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:11.597 INFO:teuthology.orchestra.run.trial134.stderr:==39996==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:11.653 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:11,652 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:11.653 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:11,652 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:11.653 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:11,652 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:11.653 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:11,652 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-25T08:40:11.653 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-25 08:40:11,652 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:13.706 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:13.707 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-25T08:40:13.707 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-25T08:40:13.707 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:13.707 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:13.707 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:40:13.727 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:13.727 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:13.775 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:13.775 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:13.782 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:13,781 seastar - Reactor backend: linux-aio 2026-02-25T08:40:13.796 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:13,795 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:13.796 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:13,795 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:13.799 INFO:teuthology.orchestra.run.trial134.stderr:==39994==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:13.799 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:13,799 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:13.860 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-25 08:40:13,859 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:13.940 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcc280b 0x10a2700 0xf47938 0x9e26106 0x9e263b3 0x9f138a5 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:13.972 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x317bbe8 0x6defb1c 0x9ef67f6 0x9ef69ac 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7b90d7c 0x7ba5769 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:14.013 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaef /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x9e6cfb9 0x9ed8b72 0x9f1a352 0x78b69db 0x7b9025f 0x7ba5769 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:14.065 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a695a0 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:14.106 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69624 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:14.123 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a6965c 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:14.163 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc08 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:14.272 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfd191 /lib64/libasan.so.8+0xef979 0x3185724 0x31858fe 0x3185bb5 0x3185b7a 0x3185caa 0x9f169a9 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:14.354 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65bc /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:14.412 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x7d56cf3 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:16.550 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:16.551 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-25T08:40:16.551 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-25T08:40:16.551 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:16.551 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:16.551 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:40:16.584 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:16.608 DEBUG:teuthology.orchestra.run.trial134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:16.760 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:16.820 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:16,820 seastar - Reactor backend: linux-aio 2026-02-25T08:40:16.831 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:16,830 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:16.831 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:16,830 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:16.833 INFO:teuthology.orchestra.run.trial134.stderr:==40057==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:16.888 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:16,887 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:16.888 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:16,887 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:16.889 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:16,887 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:16.889 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:16,887 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 2 2026-02-25T08:40:16.889 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-25 08:40:16,888 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:18.951 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:18.952 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-25T08:40:18.952 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-25T08:40:18.952 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:18.952 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:18.952 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:40:18.971 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:18.972 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:19.013 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:19.013 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:19.020 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:19,019 seastar - Reactor backend: linux-aio 2026-02-25T08:40:19.030 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:19,029 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:19.030 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:19,029 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:19.032 INFO:teuthology.orchestra.run.trial134.stderr:==40055==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:19.032 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-25 08:40:19,032 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:19.090 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-25 08:40:19,089 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:19.134 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc08 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x11380c8 0xf4792d 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e22e1c 0x9e2ae9f 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x74b5436 0x74b57bd 0x752694f 0x755c666 0x73979b2 0xa1f982a 0xa1f9a2d 0xa1f9acb 0xa1f9b50 0xa24cf52 0xa24d00b 0xa27e94d 0xa27ecf4 0xa185005 0xa1850be 0xa18514d 0xa24caba 0xa24cb7d 0xa24cc8a 0xa24ce45 0x731f233 0x742d7eb 0x72671d6 0x720df5d 0x705b52e 0x74fcd78 0x74fcecc 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:19.173 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x35fc098 0x35fc215 0x3618092 0x9e33da7 0x9e33e8a 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:19.206 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a2826 0xf47938 0x9f04791 0x9f04ad6 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7b90d7c 0x7ba5769 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:19.246 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04e6 /lib64/libasan.so.8+0xb1437 0xb7ac87 0x10a27ac 0xf47938 0x9e26106 0x9e263b3 0x9f138a5 0x9f18413 0x9ed89e1 0x9ed79e1 0x7b90d7c 0x7ba5769 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:19.299 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:19.340 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x7d56ccd 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:19.357 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129749 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:19.397 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:19.506 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x3617fd7 0x9e33da7 0x9e33e69 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:19.589 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e8f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:21.826 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:21.826 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-25T08:40:21.826 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-25T08:40:21.826 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:21.826 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-25T08:40:21.826 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:40:21.858 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:21.882 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T08:40:21.883 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:21.883 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T08:40:21.946 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:21.946 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T08:40:22.010 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:22.011 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T08:40:22.074 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:22.074 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T08:40:22.138 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:22.138 DEBUG:teuthology.orchestra.run.trial134:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T08:40:22.193 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T08:40:22.193 DEBUG:teuthology.orchestra.run.trial134:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T08:40:22.255 INFO:teuthology.orchestra.run.trial134.stdout:[mgr.x] 2026-02-25T08:40:22.255 INFO:teuthology.orchestra.run.trial134.stdout: key = AQBjtZ5pMeSSHhAA0VZhdiuYNNRsbpzkmi4CSQ== 2026-02-25T08:40:22.255 INFO:teuthology.orchestra.run.trial134.stdout:[osd.0] 2026-02-25T08:40:22.255 INFO:teuthology.orchestra.run.trial134.stdout: key = AQBotZ5priqFIRAANTjcWKX4IZE7rt+VzxkCtQ== 2026-02-25T08:40:22.255 INFO:teuthology.orchestra.run.trial134.stdout:[osd.1] 2026-02-25T08:40:22.256 INFO:teuthology.orchestra.run.trial134.stdout: key = AQBttZ5pEuF0MxAAqii/RjtdXgVthDFcCJpLGA== 2026-02-25T08:40:22.256 INFO:teuthology.orchestra.run.trial134.stdout:[osd.2] 2026-02-25T08:40:22.256 INFO:teuthology.orchestra.run.trial134.stdout: key = AQBztZ5pEMOPBRAAXT7EHDIWjGDpeesJxHD2Ng== 2026-02-25T08:40:22.256 INFO:teuthology.orchestra.run.trial134.stdout:[client.0] 2026-02-25T08:40:22.256 INFO:teuthology.orchestra.run.trial134.stdout: key = AQBjtZ5pMA9MIRAA7DqgmBFKxlatCh7KCQCRsg== 2026-02-25T08:40:22.257 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T08:40:22.346 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:40:22.394 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:40:22.442 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:40:22.492 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:40:22.543 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T08:40:22.543 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T08:40:22.567 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:22.653 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:22.651+0000 7f4499487c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:22.653 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:22.651+0000 7f4499487c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:22.664 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T08:40:22.689 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-25T08:40:22.753 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:22.838 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:22.836+0000 7f75b6ee5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:22.838 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:22.836+0000 7f75b6ee5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:22.849 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-25T08:40:22.874 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-25T08:40:22.939 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:23.023 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:23.022+0000 7fb77d845c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.024 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:23.022+0000 7fb77d845c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.035 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-25T08:40:23.099 DEBUG:teuthology.orchestra.run.trial134:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:23.156 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T08:40:23.156 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T08:40:23.157 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T08:40:23.198 INFO:tasks.ceph.mon.a:Started 2026-02-25T08:40:23.198 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-25T08:40:23.198 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-25T08:40:23.200 INFO:tasks.ceph.mon.b:Started 2026-02-25T08:40:23.201 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-25T08:40:23.201 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-25T08:40:23.203 INFO:tasks.ceph.mon.c:Started 2026-02-25T08:40:23.203 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph config set mgr mgr_pool false --force 2026-02-25T08:40:23.262 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-25T08:40:23.260+0000 7fa145706c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.262 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-25T08:40:23.260+0000 7fa145706c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.262 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-25T08:40:23.260+0000 7fa145706c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.267 INFO:tasks.ceph.mon.b.trial134.stderr:2026-02-25T08:40:23.265+0000 7f4169c2dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.267 INFO:tasks.ceph.mon.b.trial134.stderr:2026-02-25T08:40:23.265+0000 7f4169c2dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.267 INFO:tasks.ceph.mon.b.trial134.stderr:2026-02-25T08:40:23.265+0000 7f4169c2dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.270 INFO:tasks.ceph.mon.c.trial134.stderr:2026-02-25T08:40:23.268+0000 7f1c4a492c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.270 INFO:tasks.ceph.mon.c.trial134.stderr:2026-02-25T08:40:23.268+0000 7f1c4a492c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.270 INFO:tasks.ceph.mon.c.trial134.stderr:2026-02-25T08:40:23.268+0000 7f1c4a492c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:23.287 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:23.281+0000 7f6a0df8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.291 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:29.289+0000 7f6a0df8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.335 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T08:40:29.335 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T08:40:29.336 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T08:40:29.338 INFO:tasks.ceph.mgr.x:Started 2026-02-25T08:40:29.338 DEBUG:tasks.ceph:set 0 configs 2026-02-25T08:40:29.338 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph config dump 2026-02-25T08:40:29.398 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-25T08:40:29.396+0000 7f2345642fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.401 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-25T08:40:29.399+0000 7f2345642fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.401 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-25T08:40:29.399+0000 7f2345642fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.417 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:29.414+0000 7f93ebbdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.420 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:29.418+0000 7f93ebbdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.453 INFO:teuthology.orchestra.run.trial134.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T08:40:29.453 INFO:teuthology.orchestra.run.trial134.stdout:mgr dev mgr_pool false * 2026-02-25T08:40:29.462 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T08:40:29.462 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T08:40:29.538 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:29.533+0000 7f5ac4035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.540 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:29.539+0000 7f5ac4035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.575 INFO:teuthology.orchestra.run.trial134.stderr:adjusted tunables profile to default 2026-02-25T08:40:29.585 INFO:tasks.ceph:check_enable_crimson: True 2026-02-25T08:40:29.585 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-25T08:40:29.585 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-25T08:40:29.658 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:29.654+0000 7fe4bc8c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.661 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:29.659+0000 7fe4bc8c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:29.700 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-25T08:40:29.698+0000 7fa13d9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-25T08:40:29.700 INFO:tasks.ceph.mon.a.trial134.stderr:Please be aware that this feature is experimental, untested, 2026-02-25T08:40:29.700 INFO:tasks.ceph.mon.a.trial134.stderr:unsupported, and may result in data corruption, data loss, 2026-02-25T08:40:29.700 INFO:tasks.ceph.mon.a.trial134.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-25T08:40:29.700 INFO:tasks.ceph.mon.a.trial134.stderr:feature with important data. 2026-02-25T08:40:29.700 INFO:tasks.ceph.mon.a.trial134.stderr: 2026-02-25T08:40:30.243 INFO:tasks.ceph.mgr.x.trial134.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-25T08:40:30.243 INFO:tasks.ceph.mgr.x.trial134.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-25T08:40:30.243 INFO:tasks.ceph.mgr.x.trial134.stderr: from numpy import show_config as show_numpy_config 2026-02-25T08:40:30.587 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T08:40:30.588 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:30.588 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T08:40:30.614 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:30.614 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T08:40:30.678 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:40:30.678 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T08:40:30.743 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd new 2857cad8-653c-44b2-8a60-deb204e9ac6d 0 2026-02-25T08:40:30.856 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:30.851+0000 7f172c19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:30.859 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:30.857+0000 7f172c19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:30.899 INFO:teuthology.orchestra.run.trial134.stdout:0 2026-02-25T08:40:30.908 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd new 9cba2f6d-21dc-4335-b5a1-c09a81ec73ad 1 2026-02-25T08:40:30.983 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:30.978+0000 7f551c50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:30.985 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:30.984+0000 7f551c50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:31.025 INFO:teuthology.orchestra.run.trial134.stdout:1 2026-02-25T08:40:31.034 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd new 81938fc4-ebbd-4b12-a465-8fa2a9121c0b 2 2026-02-25T08:40:31.110 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:31.104+0000 7ff01da11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:31.112 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:31.111+0000 7ff01da11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:31.152 INFO:teuthology.orchestra.run.trial134.stdout:2 2026-02-25T08:40:31.160 DEBUG:teuthology.orchestra.run.trial134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-25T08:40:31.190 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T08:40:31.190 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T08:40:31.233 INFO:tasks.ceph.osd.0:Started 2026-02-25T08:40:31.233 DEBUG:teuthology.orchestra.run.trial134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-25T08:40:31.263 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T08:40:31.263 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T08:40:31.305 INFO:tasks.ceph.osd.1:Started 2026-02-25T08:40:31.305 DEBUG:teuthology.orchestra.run.trial134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-25T08:40:31.334 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T08:40:31.334 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T08:40:31.377 INFO:tasks.ceph.osd.2:Started 2026-02-25T08:40:31.378 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:40:31.455 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:31.451+0000 7f1d13b62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:31.458 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:31.457+0000 7f1d13b62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:31.492 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:31.492 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":6,"fsid":"e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b","created":"2026-02-25T08:40:28.486432+0000","modified":"2026-02-25T08:40:31.149515+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:40:31.149515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2857cad8-653c-44b2-8a60-deb204e9ac6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cba2f6d-21dc-4335-b5a1-c09a81ec73ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"81938fc4-ebbd-4b12-a465-8fa2a9121c0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:40:31.501 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T08:40:31.501 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T08:40:31.803 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:40:31.869 INFO:teuthology.misc.health.trial134.stderr:2026-02-25T08:40:31.865+0000 7f4f03a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:31.871 INFO:teuthology.misc.health.trial134.stderr:2026-02-25T08:40:31.870+0000 7f4f03a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:31.905 INFO:teuthology.misc.health.trial134.stdout: 2026-02-25T08:40:31.905 INFO:teuthology.misc.health.trial134.stdout:{"epoch":6,"fsid":"e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b","created":"2026-02-25T08:40:28.486432+0000","modified":"2026-02-25T08:40:31.149515+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:40:31.149515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2857cad8-653c-44b2-8a60-deb204e9ac6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cba2f6d-21dc-4335-b5a1-c09a81ec73ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"81938fc4-ebbd-4b12-a465-8fa2a9121c0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:40:31.913 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-25T08:40:32.629 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-25T08:40:32.628+0000 7f226025e640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-25T08:40:32.630 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-25T08:40:32.628+0000 7f226025e640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-25T08:40:38.215 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:40:38.288 INFO:teuthology.misc.health.trial134.stderr:2026-02-25T08:40:38.284+0000 7fdc18e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:38.291 INFO:teuthology.misc.health.trial134.stderr:2026-02-25T08:40:38.290+0000 7fdc18e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:38.410 INFO:teuthology.misc.health.trial134.stdout: 2026-02-25T08:40:38.410 INFO:teuthology.misc.health.trial134.stdout:{"epoch":8,"fsid":"e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b","created":"2026-02-25T08:40:28.486432+0000","modified":"2026-02-25T08:40:38.350403+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:40:31.149515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2857cad8-653c-44b2-8a60-deb204e9ac6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cba2f6d-21dc-4335-b5a1-c09a81ec73ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"81938fc4-ebbd-4b12-a465-8fa2a9121c0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:40:38.421 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-25T08:40:44.724 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:40:44.793 INFO:teuthology.misc.health.trial134.stderr:2026-02-25T08:40:44.789+0000 7f884e90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:44.795 INFO:teuthology.misc.health.trial134.stderr:2026-02-25T08:40:44.793+0000 7f884e90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:44.907 INFO:teuthology.misc.health.trial134.stdout: 2026-02-25T08:40:44.907 INFO:teuthology.misc.health.trial134.stdout:{"epoch":10,"fsid":"e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b","created":"2026-02-25T08:40:28.486432+0000","modified":"2026-02-25T08:40:40.360015+0000","last_up_change":"2026-02-25T08:40:40.360015+0000","last_in_change":"2026-02-25T08:40:31.149515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2857cad8-653c-44b2-8a60-deb204e9ac6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6807","nonce":3119415547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6804","nonce":3119415547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6811","nonce":3119415547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6810","nonce":3119415547}]},"public_addr":"10.20.193.134:6807/3119415547","cluster_addr":"10.20.193.134:6804/3119415547","heartbeat_back_addr":"10.20.193.134:6811/3119415547","heartbeat_front_addr":"10.20.193.134:6810/3119415547","state":["exists","up"]},{"osd":1,"uuid":"9cba2f6d-21dc-4335-b5a1-c09a81ec73ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6803","nonce":3523073085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6802","nonce":3523073085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6808","nonce":3523073085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6809","nonce":3523073085}]},"public_addr":"10.20.193.134:6803/3523073085","cluster_addr":"10.20.193.134:6802/3523073085","heartbeat_back_addr":"10.20.193.134:6808/3523073085","heartbeat_front_addr":"10.20.193.134:6809/3523073085","state":["exists","up"]},{"osd":2,"uuid":"81938fc4-ebbd-4b12-a465-8fa2a9121c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6805","nonce":3377902204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6806","nonce":3377902204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6815","nonce":3377902204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6814","nonce":3377902204}]},"public_addr":"10.20.193.134:6805/3377902204","cluster_addr":"10.20.193.134:6806/3377902204","heartbeat_back_addr":"10.20.193.134:6815/3377902204","heartbeat_front_addr":"10.20.193.134:6814/3377902204","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:40:44.916 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-25T08:40:44.917 INFO:tasks.ceph:Creating RBD pool 2026-02-25T08:40:44.917 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T08:40:44.992 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:44.987+0000 7fe528ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:44.994 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:44.993+0000 7fe528ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:45.963 INFO:teuthology.orchestra.run.trial134.stderr:pool 'rbd' created 2026-02-25T08:40:45.976 DEBUG:teuthology.orchestra.run.trial134:> rbd --cluster ceph pool init rbd 2026-02-25T08:40:46.017 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:46.015+0000 7f51af3ee240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:46.017 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:46.015+0000 7f51af3ee240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:46.021 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:46.019+0000 7f51af3ee240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.003 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T08:40:49.004 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T08:40:49.005 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T08:40:49.120 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.116+0000 7fa33c8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.122 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.121+0000 7fa33c8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.237 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:49.246 INFO:teuthology.orchestra.run.trial134.stdout:[{"version":2,"timestamp":"2026-02-25T08:40:29.324077+0000","name":"","changes":[{"name":"mgr/mgr_pool","new_value":"false"}]}] 2026-02-25T08:40:49.246 INFO:tasks.ceph_manager:config epoch is 2 2026-02-25T08:40:49.246 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T08:40:49.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T08:40:49.247 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T08:40:49.323 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.319+0000 7f50e0c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.326 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.324+0000 7f50e0c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.468 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:49.478 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":4,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6800","nonce":1571994039},{"type":"v1","addr":"10.20.193.134:6801","nonce":1571994039}]},"active_addr":"10.20.193.134:6801/1571994039","active_change":"2026-02-25T08:40:31.620892+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.134:0","nonce":3040349308}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.134:0","nonce":2325770222}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.134:0","nonce":2385498422}]}]} 2026-02-25T08:40:49.482 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T08:40:49.482 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:40:49.482 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:40:49.559 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.556+0000 7f38ee7a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.562 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.560+0000 7f38ee7a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.672 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:49.672 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":14,"fsid":"e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b","created":"2026-02-25T08:40:28.486432+0000","modified":"2026-02-25T08:40:48.971181+0000","last_up_change":"2026-02-25T08:40:40.360015+0000","last_in_change":"2026-02-25T08:40:31.149515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T08:40:45.107254+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2857cad8-653c-44b2-8a60-deb204e9ac6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6807","nonce":3119415547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6804","nonce":3119415547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6811","nonce":3119415547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6810","nonce":3119415547}]},"public_addr":"10.20.193.134:6807/3119415547","cluster_addr":"10.20.193.134:6804/3119415547","heartbeat_back_addr":"10.20.193.134:6811/3119415547","heartbeat_front_addr":"10.20.193.134:6810/3119415547","state":["exists","up"]},{"osd":1,"uuid":"9cba2f6d-21dc-4335-b5a1-c09a81ec73ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6803","nonce":3523073085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6802","nonce":3523073085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6808","nonce":3523073085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6809","nonce":3523073085}]},"public_addr":"10.20.193.134:6803/3523073085","cluster_addr":"10.20.193.134:6802/3523073085","heartbeat_back_addr":"10.20.193.134:6808/3523073085","heartbeat_front_addr":"10.20.193.134:6809/3523073085","state":["exists","up"]},{"osd":2,"uuid":"81938fc4-ebbd-4b12-a465-8fa2a9121c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6805","nonce":3377902204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6806","nonce":3377902204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6815","nonce":3377902204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6814","nonce":3377902204}]},"public_addr":"10.20.193.134:6805/3377902204","cluster_addr":"10.20.193.134:6806/3377902204","heartbeat_back_addr":"10.20.193.134:6815/3377902204","heartbeat_front_addr":"10.20.193.134:6814/3377902204","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:40:49.682 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T08:40:49.682 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:40:49.759 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.755+0000 7fcaeb446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.761 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.760+0000 7fcaeb446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.875 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:49.875 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":14,"fsid":"e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b","created":"2026-02-25T08:40:28.486432+0000","modified":"2026-02-25T08:40:48.971181+0000","last_up_change":"2026-02-25T08:40:40.360015+0000","last_in_change":"2026-02-25T08:40:31.149515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T08:40:45.107254+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2857cad8-653c-44b2-8a60-deb204e9ac6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6807","nonce":3119415547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6804","nonce":3119415547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6811","nonce":3119415547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6810","nonce":3119415547}]},"public_addr":"10.20.193.134:6807/3119415547","cluster_addr":"10.20.193.134:6804/3119415547","heartbeat_back_addr":"10.20.193.134:6811/3119415547","heartbeat_front_addr":"10.20.193.134:6810/3119415547","state":["exists","up"]},{"osd":1,"uuid":"9cba2f6d-21dc-4335-b5a1-c09a81ec73ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6803","nonce":3523073085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6802","nonce":3523073085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6808","nonce":3523073085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6809","nonce":3523073085}]},"public_addr":"10.20.193.134:6803/3523073085","cluster_addr":"10.20.193.134:6802/3523073085","heartbeat_back_addr":"10.20.193.134:6808/3523073085","heartbeat_front_addr":"10.20.193.134:6809/3523073085","state":["exists","up"]},{"osd":2,"uuid":"81938fc4-ebbd-4b12-a465-8fa2a9121c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6805","nonce":3377902204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6806","nonce":3377902204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6815","nonce":3377902204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6814","nonce":3377902204}]},"public_addr":"10.20.193.134:6805/3377902204","cluster_addr":"10.20.193.134:6806/3377902204","heartbeat_back_addr":"10.20.193.134:6815/3377902204","heartbeat_front_addr":"10.20.193.134:6814/3377902204","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:40:49.887 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T08:40:49.887 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T08:40:49.888 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T08:40:49.970 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.965+0000 7efc33fb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.971 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.966+0000 7f03ca90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.972 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.967+0000 7ff181574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.973 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.972+0000 7efc33fb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.974 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.972+0000 7f03ca90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.975 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:49.973+0000 7ff181574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:49.991 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-25T08:40:49.991 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:40:49.993 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-25T08:40:49.993 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:40:49.995 INFO:teuthology.orchestra.run.trial134.stdout:38654705674 2026-02-25T08:40:49.995 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:40:50.070 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:50.066+0000 7f4346bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:50.073 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:50.071+0000 7f4346bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:50.115 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:50.110+0000 7f6939d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:50.115 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:50.110+0000 7ff304983640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:50.118 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:50.116+0000 7ff304983640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:50.118 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:50.116+0000 7f6939d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:50.184 INFO:teuthology.orchestra.run.trial134.stdout:42949672964 2026-02-25T08:40:50.195 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.0 2026-02-25T08:40:50.229 INFO:teuthology.orchestra.run.trial134.stdout:38654705673 2026-02-25T08:40:50.231 INFO:teuthology.orchestra.run.trial134.stdout:42949672964 2026-02-25T08:40:50.240 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.1 2026-02-25T08:40:50.242 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.2 2026-02-25T08:40:51.196 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:40:51.240 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:40:51.243 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:40:51.276 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:51.271+0000 7f35cd4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.279 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:51.278+0000 7f35cd4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.317 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:51.312+0000 7f478743d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.320 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:51.318+0000 7f478743d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.353 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:51.349+0000 7f486cd74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.356 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:51.355+0000 7f486cd74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.392 INFO:teuthology.orchestra.run.trial134.stdout:42949672964 2026-02-25T08:40:51.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.0 2026-02-25T08:40:51.432 INFO:teuthology.orchestra.run.trial134.stdout:38654705673 2026-02-25T08:40:51.443 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.1 2026-02-25T08:40:51.467 INFO:teuthology.orchestra.run.trial134.stdout:42949672964 2026-02-25T08:40:51.477 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.2 2026-02-25T08:40:52.404 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:40:52.445 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:40:52.478 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:40:52.486 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.481+0000 7fc2938cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.488 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.487+0000 7fc2938cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.525 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.520+0000 7f73e745a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.527 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.526+0000 7f73e745a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.563 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.558+0000 7f3ba0a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.567 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.565+0000 7f3ba0a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.603 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-25T08:40:52.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-25T08:40:52.614 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:52.639 INFO:teuthology.orchestra.run.trial134.stdout:38654705674 2026-02-25T08:40:52.650 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-25T08:40:52.650 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:52.677 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-25T08:40:52.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-25T08:40:52.687 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:52.687 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:40:52.688 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:40:52.805 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.801+0000 7f5dfa30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.807 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.806+0000 7f5dfa30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:52.916 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:52.917 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-25T08:40:52.925 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-25T08:40:51.626991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":1060,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892124,"statfs":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000242"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.989140+0000","last_change":"2026-02-25T08:40:48.994930+0000","last_active":"2026-02-25T08:40:48.989140+0000","last_peered":"2026-02-25T08:40:48.989140+0000","last_clean":"2026-02-25T08:40:48.989140+0000","last_became_active":"2026-02-25T08:40:47.012178+0000","last_became_peered":"2026-02-25T08:40:47.012178+0000","last_unstale":"2026-02-25T08:40:48.989140+0000","last_undegraded":"2026-02-25T08:40:48.989140+0000","last_fullsized":"2026-02-25T08:40:48.989140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.987350+0000","last_change":"2026-02-25T08:40:48.990759+0000","last_active":"2026-02-25T08:40:48.987350+0000","last_peered":"2026-02-25T08:40:48.987350+0000","last_clean":"2026-02-25T08:40:48.987350+0000","last_became_active":"2026-02-25T08:40:47.036937+0000","last_became_peered":"2026-02-25T08:40:47.036937+0000","last_unstale":"2026-02-25T08:40:48.987350+0000","last_undegraded":"2026-02-25T08:40:48.987350+0000","last_fullsized":"2026-02-25T08:40:48.987350+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988127+0000","last_change":"2026-02-25T08:40:48.990244+0000","last_active":"2026-02-25T08:40:48.988127+0000","last_peered":"2026-02-25T08:40:48.988127+0000","last_clean":"2026-02-25T08:40:48.988127+0000","last_became_active":"2026-02-25T08:40:47.016561+0000","last_became_peered":"2026-02-25T08:40:47.016561+0000","last_unstale":"2026-02-25T08:40:48.988127+0000","last_undegraded":"2026-02-25T08:40:48.988127+0000","last_fullsized":"2026-02-25T08:40:48.988127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.990925+0000","last_change":"2026-02-25T08:40:48.995058+0000","last_active":"2026-02-25T08:40:48.990925+0000","last_peered":"2026-02-25T08:40:48.990925+0000","last_clean":"2026-02-25T08:40:48.990925+0000","last_became_active":"2026-02-25T08:40:47.060230+0000","last_became_peered":"2026-02-25T08:40:47.060230+0000","last_unstale":"2026-02-25T08:40:48.990925+0000","last_undegraded":"2026-02-25T08:40:48.990925+0000","last_fullsized":"2026-02-25T08:40:48.990925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988748+0000","last_change":"2026-02-25T08:40:48.995100+0000","last_active":"2026-02-25T08:40:48.988748+0000","last_peered":"2026-02-25T08:40:48.988748+0000","last_clean":"2026-02-25T08:40:48.988748+0000","last_became_active":"2026-02-25T08:40:47.014376+0000","last_became_peered":"2026-02-25T08:40:47.014376+0000","last_unstale":"2026-02-25T08:40:48.988748+0000","last_undegraded":"2026-02-25T08:40:48.988748+0000","last_fullsized":"2026-02-25T08:40:48.988748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988825+0000","last_change":"2026-02-25T08:40:48.988825+0000","last_active":"2026-02-25T08:40:48.988825+0000","last_peered":"2026-02-25T08:40:48.988825+0000","last_clean":"2026-02-25T08:40:48.988825+0000","last_became_active":"2026-02-25T08:40:47.017912+0000","last_became_peered":"2026-02-25T08:40:47.017912+0000","last_unstale":"2026-02-25T08:40:48.988825+0000","last_undegraded":"2026-02-25T08:40:48.988825+0000","last_fullsized":"2026-02-25T08:40:48.988825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986743+0000","last_change":"2026-02-25T08:40:48.989629+0000","last_active":"2026-02-25T08:40:48.986743+0000","last_peered":"2026-02-25T08:40:48.986743+0000","last_clean":"2026-02-25T08:40:48.986743+0000","last_became_active":"2026-02-25T08:40:47.003694+0000","last_became_peered":"2026-02-25T08:40:47.003694+0000","last_unstale":"2026-02-25T08:40:48.986743+0000","last_undegraded":"2026-02-25T08:40:48.986743+0000","last_fullsized":"2026-02-25T08:40:48.986743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986167+0000","last_change":"2026-02-25T08:40:48.990710+0000","last_active":"2026-02-25T08:40:48.986167+0000","last_peered":"2026-02-25T08:40:48.986167+0000","last_clean":"2026-02-25T08:40:48.986167+0000","last_became_active":"2026-02-25T08:40:47.046108+0000","last_became_peered":"2026-02-25T08:40:47.046108+0000","last_unstale":"2026-02-25T08:40:48.986167+0000","last_undegraded":"2026-02-25T08:40:48.986167+0000","last_fullsized":"2026-02-25T08:40:48.986167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297420,"statfs":{"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:40:52.926 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:40:53.003 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:52.999+0000 7ffb5c90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:53.006 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:53.004+0000 7ffb5c90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:53.116 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:53.117 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-25T08:40:53.126 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-25T08:40:51.626991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":1060,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892124,"statfs":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000242"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.989140+0000","last_change":"2026-02-25T08:40:48.994930+0000","last_active":"2026-02-25T08:40:48.989140+0000","last_peered":"2026-02-25T08:40:48.989140+0000","last_clean":"2026-02-25T08:40:48.989140+0000","last_became_active":"2026-02-25T08:40:47.012178+0000","last_became_peered":"2026-02-25T08:40:47.012178+0000","last_unstale":"2026-02-25T08:40:48.989140+0000","last_undegraded":"2026-02-25T08:40:48.989140+0000","last_fullsized":"2026-02-25T08:40:48.989140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.987350+0000","last_change":"2026-02-25T08:40:48.990759+0000","last_active":"2026-02-25T08:40:48.987350+0000","last_peered":"2026-02-25T08:40:48.987350+0000","last_clean":"2026-02-25T08:40:48.987350+0000","last_became_active":"2026-02-25T08:40:47.036937+0000","last_became_peered":"2026-02-25T08:40:47.036937+0000","last_unstale":"2026-02-25T08:40:48.987350+0000","last_undegraded":"2026-02-25T08:40:48.987350+0000","last_fullsized":"2026-02-25T08:40:48.987350+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988127+0000","last_change":"2026-02-25T08:40:48.990244+0000","last_active":"2026-02-25T08:40:48.988127+0000","last_peered":"2026-02-25T08:40:48.988127+0000","last_clean":"2026-02-25T08:40:48.988127+0000","last_became_active":"2026-02-25T08:40:47.016561+0000","last_became_peered":"2026-02-25T08:40:47.016561+0000","last_unstale":"2026-02-25T08:40:48.988127+0000","last_undegraded":"2026-02-25T08:40:48.988127+0000","last_fullsized":"2026-02-25T08:40:48.988127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.990925+0000","last_change":"2026-02-25T08:40:48.995058+0000","last_active":"2026-02-25T08:40:48.990925+0000","last_peered":"2026-02-25T08:40:48.990925+0000","last_clean":"2026-02-25T08:40:48.990925+0000","last_became_active":"2026-02-25T08:40:47.060230+0000","last_became_peered":"2026-02-25T08:40:47.060230+0000","last_unstale":"2026-02-25T08:40:48.990925+0000","last_undegraded":"2026-02-25T08:40:48.990925+0000","last_fullsized":"2026-02-25T08:40:48.990925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988748+0000","last_change":"2026-02-25T08:40:48.995100+0000","last_active":"2026-02-25T08:40:48.988748+0000","last_peered":"2026-02-25T08:40:48.988748+0000","last_clean":"2026-02-25T08:40:48.988748+0000","last_became_active":"2026-02-25T08:40:47.014376+0000","last_became_peered":"2026-02-25T08:40:47.014376+0000","last_unstale":"2026-02-25T08:40:48.988748+0000","last_undegraded":"2026-02-25T08:40:48.988748+0000","last_fullsized":"2026-02-25T08:40:48.988748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988825+0000","last_change":"2026-02-25T08:40:48.988825+0000","last_active":"2026-02-25T08:40:48.988825+0000","last_peered":"2026-02-25T08:40:48.988825+0000","last_clean":"2026-02-25T08:40:48.988825+0000","last_became_active":"2026-02-25T08:40:47.017912+0000","last_became_peered":"2026-02-25T08:40:47.017912+0000","last_unstale":"2026-02-25T08:40:48.988825+0000","last_undegraded":"2026-02-25T08:40:48.988825+0000","last_fullsized":"2026-02-25T08:40:48.988825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986743+0000","last_change":"2026-02-25T08:40:48.989629+0000","last_active":"2026-02-25T08:40:48.986743+0000","last_peered":"2026-02-25T08:40:48.986743+0000","last_clean":"2026-02-25T08:40:48.986743+0000","last_became_active":"2026-02-25T08:40:47.003694+0000","last_became_peered":"2026-02-25T08:40:47.003694+0000","last_unstale":"2026-02-25T08:40:48.986743+0000","last_undegraded":"2026-02-25T08:40:48.986743+0000","last_fullsized":"2026-02-25T08:40:48.986743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986167+0000","last_change":"2026-02-25T08:40:48.990710+0000","last_active":"2026-02-25T08:40:48.986167+0000","last_peered":"2026-02-25T08:40:48.986167+0000","last_clean":"2026-02-25T08:40:48.986167+0000","last_became_active":"2026-02-25T08:40:47.046108+0000","last_became_peered":"2026-02-25T08:40:47.046108+0000","last_unstale":"2026-02-25T08:40:48.986167+0000","last_undegraded":"2026-02-25T08:40:48.986167+0000","last_fullsized":"2026-02-25T08:40:48.986167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297420,"statfs":{"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:40:53.127 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:40:53.127 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T08:40:53.127 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T08:40:53.127 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T08:40:53.201 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:53.198+0000 7fcd062cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:53.203 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:40:53.202+0000 7fcd062cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:53.343 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:40:53.343 INFO:teuthology.orchestra.run.trial134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T08:40:53.353 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T08:40:53.353 INFO:teuthology.run_tasks:Running task workunit... 2026-02-25T08:40:53.379 INFO:tasks.workunit:Pulling workunits from ref c32469671c6b0222f786dd36c34eb702f780a11d 2026-02-25T08:40:53.379 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-25T08:40:53.379 DEBUG:teuthology.orchestra.run.trial134:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-25T08:40:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:40:53.396 INFO:teuthology.orchestra.run.trial134.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-25T08:40:53.397 DEBUG:teuthology.orchestra.run.trial134:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T08:40:53.452 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-25T08:40:53.453 DEBUG:teuthology.orchestra.run.trial134:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-25T08:40:53.509 INFO:tasks.workunit:timeout=3h 2026-02-25T08:40:53.509 INFO:tasks.workunit:cleanup=True 2026-02-25T08:40:53.510 DEBUG:teuthology.orchestra.run.trial134:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c32469671c6b0222f786dd36c34eb702f780a11d 2026-02-25T08:40:53.573 INFO:tasks.workunit.client.0.trial134.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-25T08:41:17.092 INFO:tasks.workunit.client.0.trial134.stderr:Note: switching to 'c32469671c6b0222f786dd36c34eb702f780a11d'. 2026-02-25T08:41:17.092 INFO:tasks.workunit.client.0.trial134.stderr: 2026-02-25T08:41:17.092 INFO:tasks.workunit.client.0.trial134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-25T08:41:17.092 INFO:tasks.workunit.client.0.trial134.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-25T08:41:17.092 INFO:tasks.workunit.client.0.trial134.stderr:state without impacting any branches by switching back to a branch. 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: git switch -c 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr:Or undo this operation with: 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: git switch - 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-25T08:41:17.093 INFO:tasks.workunit.client.0.trial134.stderr: 2026-02-25T08:41:17.094 INFO:tasks.workunit.client.0.trial134.stderr:HEAD is now at c32469671c6 Merge pull request #66090 from rhcs-dashboard/73674-create-rule-certmgr 2026-02-25T08:41:17.095 DEBUG:teuthology.orchestra.run.trial134:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-25T08:41:17.151 INFO:tasks.workunit.client.0.trial134.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-25T08:41:17.153 INFO:tasks.workunit.client.0.trial134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-25T08:41:17.153 INFO:tasks.workunit.client.0.trial134.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-25T08:41:17.240 INFO:tasks.workunit.client.0.trial134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-25T08:41:17.273 INFO:tasks.workunit.client.0.trial134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-25T08:41:17.300 INFO:tasks.workunit.client.0.trial134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-25T08:41:17.301 INFO:tasks.workunit.client.0.trial134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-25T08:41:17.301 INFO:tasks.workunit.client.0.trial134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-25T08:41:17.329 INFO:tasks.workunit.client.0.trial134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-25T08:41:17.332 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-25T08:41:17.333 DEBUG:teuthology.orchestra.run.trial134:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-25T08:41:17.389 INFO:tasks.workunit:Running workunits matching cephtool/test_crimson_dump_metrics.sh on client.0... 2026-02-25T08:41:17.390 INFO:tasks.workunit:Running workunit cephtool/test_crimson_dump_metrics.sh... 2026-02-25T08:41:17.391 DEBUG:teuthology.orchestra.run.trial134:workunit test cephtool/test_crimson_dump_metrics.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c32469671c6b0222f786dd36c34eb702f780a11d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh 2026-02-25T08:41:17.451 INFO:tasks.workunit.client.0.trial134.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T08:41:17.454 INFO:tasks.workunit.client.0.trial134.stderr:+ test_dump_metrics 2026-02-25T08:41:17.454 INFO:tasks.workunit.client.0.trial134.stderr:+ ceph tell osd.0 dump_metrics 2026-02-25T08:41:17.567 INFO:tasks.workunit.client.0.trial134.stderr:+ ceph tell osd.0 dump_metrics reactor_utilization 2026-02-25T08:41:17.647 INFO:tasks.workunit.client.0.trial134.stderr:++ grep -c reactor_utilization /tmp/dump_metrics_full.json 2026-02-25T08:41:17.648 INFO:tasks.workunit.client.0.trial134.stderr:+ full_count=2 2026-02-25T08:41:17.648 INFO:tasks.workunit.client.0.trial134.stderr:++ grep -c reactor_utilization /tmp/dump_metrics_reactor.json 2026-02-25T08:41:17.649 INFO:tasks.workunit.client.0.trial134.stderr:+ reactor_count=2 2026-02-25T08:41:17.649 INFO:tasks.workunit.client.0.trial134.stderr:+ '[' 2 -gt 0 ']' 2026-02-25T08:41:17.649 INFO:tasks.workunit.client.0.trial134.stderr:+ '[' 2 -gt 0 ']' 2026-02-25T08:41:17.649 INFO:tasks.workunit.client.0.trial134.stderr:+ '[' 2 -eq 2 ']' 2026-02-25T08:41:17.649 INFO:tasks.workunit.client.0.trial134.stderr:+ rm -f /tmp/dump_metrics_full.json /tmp/dump_metrics_reactor.json 2026-02-25T08:41:17.650 INFO:tasks.workunit.client.0.trial134.stderr:+ return 0 2026-02-25T08:41:17.650 INFO:tasks.workunit.client.0.trial134.stderr:+ echo OK 2026-02-25T08:41:17.650 INFO:tasks.workunit.client.0.trial134.stdout:OK 2026-02-25T08:41:17.650 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T08:41:17.650 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T08:41:17.716 INFO:tasks.workunit:Stopping ['cephtool/test_crimson_dump_metrics.sh'] on client.0... 2026-02-25T08:41:17.716 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-25T08:41:18.075 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:18.076 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T08:41:18.101 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T08:41:18.101 DEBUG:teuthology.orchestra.run.trial134:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T08:41:18.157 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T08:41:18.158 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T08:41:18.173 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:18.173 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:18.272 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.268+0000 7fee8305a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.275 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.273+0000 7fee8305a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.384 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:41:18.384 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-25T08:41:18.393 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2026-02-25T08:41:17.630640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":1060,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892124,"statfs":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001707"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.989140+0000","last_change":"2026-02-25T08:40:48.994930+0000","last_active":"2026-02-25T08:40:48.989140+0000","last_peered":"2026-02-25T08:40:48.989140+0000","last_clean":"2026-02-25T08:40:48.989140+0000","last_became_active":"2026-02-25T08:40:47.012178+0000","last_became_peered":"2026-02-25T08:40:47.012178+0000","last_unstale":"2026-02-25T08:40:48.989140+0000","last_undegraded":"2026-02-25T08:40:48.989140+0000","last_fullsized":"2026-02-25T08:40:48.989140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.987350+0000","last_change":"2026-02-25T08:40:48.990759+0000","last_active":"2026-02-25T08:40:48.987350+0000","last_peered":"2026-02-25T08:40:48.987350+0000","last_clean":"2026-02-25T08:40:48.987350+0000","last_became_active":"2026-02-25T08:40:47.036937+0000","last_became_peered":"2026-02-25T08:40:47.036937+0000","last_unstale":"2026-02-25T08:40:48.987350+0000","last_undegraded":"2026-02-25T08:40:48.987350+0000","last_fullsized":"2026-02-25T08:40:48.987350+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988127+0000","last_change":"2026-02-25T08:40:48.990244+0000","last_active":"2026-02-25T08:40:48.988127+0000","last_peered":"2026-02-25T08:40:48.988127+0000","last_clean":"2026-02-25T08:40:48.988127+0000","last_became_active":"2026-02-25T08:40:47.016561+0000","last_became_peered":"2026-02-25T08:40:47.016561+0000","last_unstale":"2026-02-25T08:40:48.988127+0000","last_undegraded":"2026-02-25T08:40:48.988127+0000","last_fullsized":"2026-02-25T08:40:48.988127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.990925+0000","last_change":"2026-02-25T08:40:48.995058+0000","last_active":"2026-02-25T08:40:48.990925+0000","last_peered":"2026-02-25T08:40:48.990925+0000","last_clean":"2026-02-25T08:40:48.990925+0000","last_became_active":"2026-02-25T08:40:47.060230+0000","last_became_peered":"2026-02-25T08:40:47.060230+0000","last_unstale":"2026-02-25T08:40:48.990925+0000","last_undegraded":"2026-02-25T08:40:48.990925+0000","last_fullsized":"2026-02-25T08:40:48.990925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988748+0000","last_change":"2026-02-25T08:40:48.995100+0000","last_active":"2026-02-25T08:40:48.988748+0000","last_peered":"2026-02-25T08:40:48.988748+0000","last_clean":"2026-02-25T08:40:48.988748+0000","last_became_active":"2026-02-25T08:40:47.014376+0000","last_became_peered":"2026-02-25T08:40:47.014376+0000","last_unstale":"2026-02-25T08:40:48.988748+0000","last_undegraded":"2026-02-25T08:40:48.988748+0000","last_fullsized":"2026-02-25T08:40:48.988748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988825+0000","last_change":"2026-02-25T08:40:48.988825+0000","last_active":"2026-02-25T08:40:48.988825+0000","last_peered":"2026-02-25T08:40:48.988825+0000","last_clean":"2026-02-25T08:40:48.988825+0000","last_became_active":"2026-02-25T08:40:47.017912+0000","last_became_peered":"2026-02-25T08:40:47.017912+0000","last_unstale":"2026-02-25T08:40:48.988825+0000","last_undegraded":"2026-02-25T08:40:48.988825+0000","last_fullsized":"2026-02-25T08:40:48.988825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986743+0000","last_change":"2026-02-25T08:40:48.989629+0000","last_active":"2026-02-25T08:40:48.986743+0000","last_peered":"2026-02-25T08:40:48.986743+0000","last_clean":"2026-02-25T08:40:48.986743+0000","last_became_active":"2026-02-25T08:40:47.003694+0000","last_became_peered":"2026-02-25T08:40:47.003694+0000","last_unstale":"2026-02-25T08:40:48.986743+0000","last_undegraded":"2026-02-25T08:40:48.986743+0000","last_fullsized":"2026-02-25T08:40:48.986743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986167+0000","last_change":"2026-02-25T08:40:48.990710+0000","last_active":"2026-02-25T08:40:48.986167+0000","last_peered":"2026-02-25T08:40:48.986167+0000","last_clean":"2026-02-25T08:40:48.986167+0000","last_became_active":"2026-02-25T08:40:47.046108+0000","last_became_peered":"2026-02-25T08:40:47.046108+0000","last_unstale":"2026-02-25T08:40:48.986167+0000","last_undegraded":"2026-02-25T08:40:48.986167+0000","last_fullsized":"2026-02-25T08:40:48.986167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297420,"statfs":{"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:18.395 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:18.470 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.465+0000 7f64fa08b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.472 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.471+0000 7f64fa08b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.582 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:41:18.582 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-25T08:41:18.591 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2026-02-25T08:41:17.630640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":1060,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892124,"statfs":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001707"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.989140+0000","last_change":"2026-02-25T08:40:48.994930+0000","last_active":"2026-02-25T08:40:48.989140+0000","last_peered":"2026-02-25T08:40:48.989140+0000","last_clean":"2026-02-25T08:40:48.989140+0000","last_became_active":"2026-02-25T08:40:47.012178+0000","last_became_peered":"2026-02-25T08:40:47.012178+0000","last_unstale":"2026-02-25T08:40:48.989140+0000","last_undegraded":"2026-02-25T08:40:48.989140+0000","last_fullsized":"2026-02-25T08:40:48.989140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.987350+0000","last_change":"2026-02-25T08:40:48.990759+0000","last_active":"2026-02-25T08:40:48.987350+0000","last_peered":"2026-02-25T08:40:48.987350+0000","last_clean":"2026-02-25T08:40:48.987350+0000","last_became_active":"2026-02-25T08:40:47.036937+0000","last_became_peered":"2026-02-25T08:40:47.036937+0000","last_unstale":"2026-02-25T08:40:48.987350+0000","last_undegraded":"2026-02-25T08:40:48.987350+0000","last_fullsized":"2026-02-25T08:40:48.987350+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988127+0000","last_change":"2026-02-25T08:40:48.990244+0000","last_active":"2026-02-25T08:40:48.988127+0000","last_peered":"2026-02-25T08:40:48.988127+0000","last_clean":"2026-02-25T08:40:48.988127+0000","last_became_active":"2026-02-25T08:40:47.016561+0000","last_became_peered":"2026-02-25T08:40:47.016561+0000","last_unstale":"2026-02-25T08:40:48.988127+0000","last_undegraded":"2026-02-25T08:40:48.988127+0000","last_fullsized":"2026-02-25T08:40:48.988127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.990925+0000","last_change":"2026-02-25T08:40:48.995058+0000","last_active":"2026-02-25T08:40:48.990925+0000","last_peered":"2026-02-25T08:40:48.990925+0000","last_clean":"2026-02-25T08:40:48.990925+0000","last_became_active":"2026-02-25T08:40:47.060230+0000","last_became_peered":"2026-02-25T08:40:47.060230+0000","last_unstale":"2026-02-25T08:40:48.990925+0000","last_undegraded":"2026-02-25T08:40:48.990925+0000","last_fullsized":"2026-02-25T08:40:48.990925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988748+0000","last_change":"2026-02-25T08:40:48.995100+0000","last_active":"2026-02-25T08:40:48.988748+0000","last_peered":"2026-02-25T08:40:48.988748+0000","last_clean":"2026-02-25T08:40:48.988748+0000","last_became_active":"2026-02-25T08:40:47.014376+0000","last_became_peered":"2026-02-25T08:40:47.014376+0000","last_unstale":"2026-02-25T08:40:48.988748+0000","last_undegraded":"2026-02-25T08:40:48.988748+0000","last_fullsized":"2026-02-25T08:40:48.988748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988825+0000","last_change":"2026-02-25T08:40:48.988825+0000","last_active":"2026-02-25T08:40:48.988825+0000","last_peered":"2026-02-25T08:40:48.988825+0000","last_clean":"2026-02-25T08:40:48.988825+0000","last_became_active":"2026-02-25T08:40:47.017912+0000","last_became_peered":"2026-02-25T08:40:47.017912+0000","last_unstale":"2026-02-25T08:40:48.988825+0000","last_undegraded":"2026-02-25T08:40:48.988825+0000","last_fullsized":"2026-02-25T08:40:48.988825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986743+0000","last_change":"2026-02-25T08:40:48.989629+0000","last_active":"2026-02-25T08:40:48.986743+0000","last_peered":"2026-02-25T08:40:48.986743+0000","last_clean":"2026-02-25T08:40:48.986743+0000","last_became_active":"2026-02-25T08:40:47.003694+0000","last_became_peered":"2026-02-25T08:40:47.003694+0000","last_unstale":"2026-02-25T08:40:48.986743+0000","last_undegraded":"2026-02-25T08:40:48.986743+0000","last_fullsized":"2026-02-25T08:40:48.986743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986167+0000","last_change":"2026-02-25T08:40:48.990710+0000","last_active":"2026-02-25T08:40:48.986167+0000","last_peered":"2026-02-25T08:40:48.986167+0000","last_clean":"2026-02-25T08:40:48.986167+0000","last_became_active":"2026-02-25T08:40:47.046108+0000","last_became_peered":"2026-02-25T08:40:47.046108+0000","last_unstale":"2026-02-25T08:40:48.986167+0000","last_undegraded":"2026-02-25T08:40:48.986167+0000","last_fullsized":"2026-02-25T08:40:48.986167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297420,"statfs":{"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:18.592 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:18.592 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:18.669 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.664+0000 7f693a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.671 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.670+0000 7f693a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.783 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:41:18.784 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-25T08:41:18.792 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2026-02-25T08:41:17.630640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":1060,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892124,"statfs":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001707"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.989140+0000","last_change":"2026-02-25T08:40:48.994930+0000","last_active":"2026-02-25T08:40:48.989140+0000","last_peered":"2026-02-25T08:40:48.989140+0000","last_clean":"2026-02-25T08:40:48.989140+0000","last_became_active":"2026-02-25T08:40:47.012178+0000","last_became_peered":"2026-02-25T08:40:47.012178+0000","last_unstale":"2026-02-25T08:40:48.989140+0000","last_undegraded":"2026-02-25T08:40:48.989140+0000","last_fullsized":"2026-02-25T08:40:48.989140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.987350+0000","last_change":"2026-02-25T08:40:48.990759+0000","last_active":"2026-02-25T08:40:48.987350+0000","last_peered":"2026-02-25T08:40:48.987350+0000","last_clean":"2026-02-25T08:40:48.987350+0000","last_became_active":"2026-02-25T08:40:47.036937+0000","last_became_peered":"2026-02-25T08:40:47.036937+0000","last_unstale":"2026-02-25T08:40:48.987350+0000","last_undegraded":"2026-02-25T08:40:48.987350+0000","last_fullsized":"2026-02-25T08:40:48.987350+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988127+0000","last_change":"2026-02-25T08:40:48.990244+0000","last_active":"2026-02-25T08:40:48.988127+0000","last_peered":"2026-02-25T08:40:48.988127+0000","last_clean":"2026-02-25T08:40:48.988127+0000","last_became_active":"2026-02-25T08:40:47.016561+0000","last_became_peered":"2026-02-25T08:40:47.016561+0000","last_unstale":"2026-02-25T08:40:48.988127+0000","last_undegraded":"2026-02-25T08:40:48.988127+0000","last_fullsized":"2026-02-25T08:40:48.988127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.990925+0000","last_change":"2026-02-25T08:40:48.995058+0000","last_active":"2026-02-25T08:40:48.990925+0000","last_peered":"2026-02-25T08:40:48.990925+0000","last_clean":"2026-02-25T08:40:48.990925+0000","last_became_active":"2026-02-25T08:40:47.060230+0000","last_became_peered":"2026-02-25T08:40:47.060230+0000","last_unstale":"2026-02-25T08:40:48.990925+0000","last_undegraded":"2026-02-25T08:40:48.990925+0000","last_fullsized":"2026-02-25T08:40:48.990925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988748+0000","last_change":"2026-02-25T08:40:48.995100+0000","last_active":"2026-02-25T08:40:48.988748+0000","last_peered":"2026-02-25T08:40:48.988748+0000","last_clean":"2026-02-25T08:40:48.988748+0000","last_became_active":"2026-02-25T08:40:47.014376+0000","last_became_peered":"2026-02-25T08:40:47.014376+0000","last_unstale":"2026-02-25T08:40:48.988748+0000","last_undegraded":"2026-02-25T08:40:48.988748+0000","last_fullsized":"2026-02-25T08:40:48.988748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988825+0000","last_change":"2026-02-25T08:40:48.988825+0000","last_active":"2026-02-25T08:40:48.988825+0000","last_peered":"2026-02-25T08:40:48.988825+0000","last_clean":"2026-02-25T08:40:48.988825+0000","last_became_active":"2026-02-25T08:40:47.017912+0000","last_became_peered":"2026-02-25T08:40:47.017912+0000","last_unstale":"2026-02-25T08:40:48.988825+0000","last_undegraded":"2026-02-25T08:40:48.988825+0000","last_fullsized":"2026-02-25T08:40:48.988825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986743+0000","last_change":"2026-02-25T08:40:48.989629+0000","last_active":"2026-02-25T08:40:48.986743+0000","last_peered":"2026-02-25T08:40:48.986743+0000","last_clean":"2026-02-25T08:40:48.986743+0000","last_became_active":"2026-02-25T08:40:47.003694+0000","last_became_peered":"2026-02-25T08:40:47.003694+0000","last_unstale":"2026-02-25T08:40:48.986743+0000","last_undegraded":"2026-02-25T08:40:48.986743+0000","last_fullsized":"2026-02-25T08:40:48.986743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986167+0000","last_change":"2026-02-25T08:40:48.990710+0000","last_active":"2026-02-25T08:40:48.986167+0000","last_peered":"2026-02-25T08:40:48.986167+0000","last_clean":"2026-02-25T08:40:48.986167+0000","last_became_active":"2026-02-25T08:40:47.046108+0000","last_became_peered":"2026-02-25T08:40:47.046108+0000","last_unstale":"2026-02-25T08:40:48.986167+0000","last_undegraded":"2026-02-25T08:40:48.986167+0000","last_fullsized":"2026-02-25T08:40:48.986167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297420,"statfs":{"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:18.794 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:18.869 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.865+0000 7f7e15b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.872 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:18.870+0000 7f7e15b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.983 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:41:18.983 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":14,"fsid":"e6f9b9b0-ff28-4ed5-8ccd-eadd712f344b","created":"2026-02-25T08:40:28.486432+0000","modified":"2026-02-25T08:40:48.971181+0000","last_up_change":"2026-02-25T08:40:40.360015+0000","last_in_change":"2026-02-25T08:40:31.149515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T08:40:45.107254+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2857cad8-653c-44b2-8a60-deb204e9ac6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6807","nonce":3119415547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6804","nonce":3119415547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6811","nonce":3119415547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6810","nonce":3119415547}]},"public_addr":"10.20.193.134:6807/3119415547","cluster_addr":"10.20.193.134:6804/3119415547","heartbeat_back_addr":"10.20.193.134:6811/3119415547","heartbeat_front_addr":"10.20.193.134:6810/3119415547","state":["exists","up"]},{"osd":1,"uuid":"9cba2f6d-21dc-4335-b5a1-c09a81ec73ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6803","nonce":3523073085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6802","nonce":3523073085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6808","nonce":3523073085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6809","nonce":3523073085}]},"public_addr":"10.20.193.134:6803/3523073085","cluster_addr":"10.20.193.134:6802/3523073085","heartbeat_back_addr":"10.20.193.134:6808/3523073085","heartbeat_front_addr":"10.20.193.134:6809/3523073085","state":["exists","up"]},{"osd":2,"uuid":"81938fc4-ebbd-4b12-a465-8fa2a9121c0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6805","nonce":3377902204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6806","nonce":3377902204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6815","nonce":3377902204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6814","nonce":3377902204}]},"public_addr":"10.20.193.134:6805/3377902204","cluster_addr":"10.20.193.134:6806/3377902204","heartbeat_back_addr":"10.20.193.134:6815/3377902204","heartbeat_front_addr":"10.20.193.134:6814/3377902204","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:19.995 INFO:tasks.ceph:Scrubbing osd.0 2026-02-25T08:41:19.995 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-25T08:41:20.075 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.068+0000 7fc2b53b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.078 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.076+0000 7fc2b53b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.088 INFO:teuthology.orchestra.run.trial134.stdout:{ 2026-02-25T08:41:20.088 INFO:teuthology.orchestra.run.trial134.stdout: "success": "" 2026-02-25T08:41:20.088 INFO:teuthology.orchestra.run.trial134.stdout:} 2026-02-25T08:41:20.095 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-25T08:41:20.170 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.166+0000 7fe33d19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.173 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.171+0000 7fe33d19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.281 INFO:teuthology.orchestra.run.trial134.stderr:instructed osd(s) 0 to deep-scrub 2026-02-25T08:41:20.291 INFO:tasks.ceph:Scrubbing osd.1 2026-02-25T08:41:20.291 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-25T08:41:20.370 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.364+0000 7fc21a10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.373 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.371+0000 7fc21a10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.382 INFO:teuthology.orchestra.run.trial134.stdout:{ 2026-02-25T08:41:20.382 INFO:teuthology.orchestra.run.trial134.stdout: "success": "" 2026-02-25T08:41:20.382 INFO:teuthology.orchestra.run.trial134.stdout:} 2026-02-25T08:41:20.389 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-25T08:41:20.466 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.462+0000 7f6df4d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.468 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.467+0000 7f6df4d9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.578 INFO:teuthology.orchestra.run.trial134.stderr:instructed osd(s) 1 to deep-scrub 2026-02-25T08:41:20.589 INFO:tasks.ceph:Scrubbing osd.2 2026-02-25T08:41:20.590 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-25T08:41:20.667 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.661+0000 7f789310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.670 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.669+0000 7f789310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.680 INFO:teuthology.orchestra.run.trial134.stdout:{ 2026-02-25T08:41:20.680 INFO:teuthology.orchestra.run.trial134.stdout: "success": "" 2026-02-25T08:41:20.680 INFO:teuthology.orchestra.run.trial134.stdout:} 2026-02-25T08:41:20.687 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-25T08:41:20.764 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.759+0000 7f113bc3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.766 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.765+0000 7f113bc3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.877 INFO:teuthology.orchestra.run.trial134.stderr:instructed osd(s) 2 to deep-scrub 2026-02-25T08:41:20.890 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:20.964 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.960+0000 7f7b96183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.966 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:20.965+0000 7f7b96183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.077 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:41:21.078 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-25T08:41:21.087 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2026-02-25T08:41:19.630935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":1060,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892124,"statfs":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001716"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.989140+0000","last_change":"2026-02-25T08:40:48.994930+0000","last_active":"2026-02-25T08:40:48.989140+0000","last_peered":"2026-02-25T08:40:48.989140+0000","last_clean":"2026-02-25T08:40:48.989140+0000","last_became_active":"2026-02-25T08:40:47.012178+0000","last_became_peered":"2026-02-25T08:40:47.012178+0000","last_unstale":"2026-02-25T08:40:48.989140+0000","last_undegraded":"2026-02-25T08:40:48.989140+0000","last_fullsized":"2026-02-25T08:40:48.989140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.987350+0000","last_change":"2026-02-25T08:40:48.990759+0000","last_active":"2026-02-25T08:40:48.987350+0000","last_peered":"2026-02-25T08:40:48.987350+0000","last_clean":"2026-02-25T08:40:48.987350+0000","last_became_active":"2026-02-25T08:40:47.036937+0000","last_became_peered":"2026-02-25T08:40:47.036937+0000","last_unstale":"2026-02-25T08:40:48.987350+0000","last_undegraded":"2026-02-25T08:40:48.987350+0000","last_fullsized":"2026-02-25T08:40:48.987350+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988127+0000","last_change":"2026-02-25T08:40:48.990244+0000","last_active":"2026-02-25T08:40:48.988127+0000","last_peered":"2026-02-25T08:40:48.988127+0000","last_clean":"2026-02-25T08:40:48.988127+0000","last_became_active":"2026-02-25T08:40:47.016561+0000","last_became_peered":"2026-02-25T08:40:47.016561+0000","last_unstale":"2026-02-25T08:40:48.988127+0000","last_undegraded":"2026-02-25T08:40:48.988127+0000","last_fullsized":"2026-02-25T08:40:48.988127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.990925+0000","last_change":"2026-02-25T08:40:48.995058+0000","last_active":"2026-02-25T08:40:48.990925+0000","last_peered":"2026-02-25T08:40:48.990925+0000","last_clean":"2026-02-25T08:40:48.990925+0000","last_became_active":"2026-02-25T08:40:47.060230+0000","last_became_peered":"2026-02-25T08:40:47.060230+0000","last_unstale":"2026-02-25T08:40:48.990925+0000","last_undegraded":"2026-02-25T08:40:48.990925+0000","last_fullsized":"2026-02-25T08:40:48.990925+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988748+0000","last_change":"2026-02-25T08:40:48.995100+0000","last_active":"2026-02-25T08:40:48.988748+0000","last_peered":"2026-02-25T08:40:48.988748+0000","last_clean":"2026-02-25T08:40:48.988748+0000","last_became_active":"2026-02-25T08:40:47.014376+0000","last_became_peered":"2026-02-25T08:40:47.014376+0000","last_unstale":"2026-02-25T08:40:48.988748+0000","last_undegraded":"2026-02-25T08:40:48.988748+0000","last_fullsized":"2026-02-25T08:40:48.988748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.988825+0000","last_change":"2026-02-25T08:40:48.988825+0000","last_active":"2026-02-25T08:40:48.988825+0000","last_peered":"2026-02-25T08:40:48.988825+0000","last_clean":"2026-02-25T08:40:48.988825+0000","last_became_active":"2026-02-25T08:40:47.017912+0000","last_became_peered":"2026-02-25T08:40:47.017912+0000","last_unstale":"2026-02-25T08:40:48.988825+0000","last_undegraded":"2026-02-25T08:40:48.988825+0000","last_fullsized":"2026-02-25T08:40:48.988825+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986743+0000","last_change":"2026-02-25T08:40:48.989629+0000","last_active":"2026-02-25T08:40:48.986743+0000","last_peered":"2026-02-25T08:40:48.986743+0000","last_clean":"2026-02-25T08:40:48.986743+0000","last_became_active":"2026-02-25T08:40:47.003694+0000","last_became_peered":"2026-02-25T08:40:47.003694+0000","last_unstale":"2026-02-25T08:40:48.986743+0000","last_undegraded":"2026-02-25T08:40:48.986743+0000","last_fullsized":"2026-02-25T08:40:48.986743+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:40:48.986167+0000","last_change":"2026-02-25T08:40:48.990710+0000","last_active":"2026-02-25T08:40:48.986167+0000","last_peered":"2026-02-25T08:40:48.986167+0000","last_clean":"2026-02-25T08:40:48.986167+0000","last_became_active":"2026-02-25T08:40:47.046108+0000","last_became_peered":"2026-02-25T08:40:47.046108+0000","last_unstale":"2026-02-25T08:40:48.986167+0000","last_undegraded":"2026-02-25T08:40:48.986167+0000","last_fullsized":"2026-02-25T08:40:48.986167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:40:45.957859+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297420,"statfs":{"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-25T08:40:45.957859+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=40, tm_sec=45, tm_wday=2, tm_yday=56, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=25, tm_hour=8, tm_min=41, tm_sec=18, tm_wday=2, tm_yday=56, tm_isdst=0) 2026-02-25T08:41:21.090 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-25T08:41:41.091 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:41.168 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:41.164+0000 7facf87d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:41.171 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:41.169+0000 7facf87d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:41.280 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-25T08:41:41.280 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-25T08:41:41.289 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2026-02-25T08:41:39.634622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":1060,"kb_used_data":1060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156892124,"statfs":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002116"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.581293+0000","last_change":"2026-02-25T08:41:20.591915+0000","last_active":"2026-02-25T08:41:20.581293+0000","last_peered":"2026-02-25T08:41:20.581293+0000","last_clean":"2026-02-25T08:41:20.581293+0000","last_became_active":"2026-02-25T08:40:47.012178+0000","last_became_peered":"2026-02-25T08:40:47.012178+0000","last_unstale":"2026-02-25T08:41:20.581293+0000","last_undegraded":"2026-02-25T08:41:20.581293+0000","last_fullsized":"2026-02-25T08:41:20.581293+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.591856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.591856+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.580482+0000","last_change":"2026-02-25T08:41:20.588892+0000","last_active":"2026-02-25T08:41:20.580482+0000","last_peered":"2026-02-25T08:41:20.580482+0000","last_clean":"2026-02-25T08:41:20.580482+0000","last_became_active":"2026-02-25T08:40:47.036937+0000","last_became_peered":"2026-02-25T08:40:47.036937+0000","last_unstale":"2026-02-25T08:41:20.580482+0000","last_undegraded":"2026-02-25T08:41:20.580482+0000","last_fullsized":"2026-02-25T08:41:20.580482+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.588805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.588805+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.878402+0000","last_change":"2026-02-25T08:41:20.887772+0000","last_active":"2026-02-25T08:41:20.878402+0000","last_peered":"2026-02-25T08:41:20.878402+0000","last_clean":"2026-02-25T08:41:20.878402+0000","last_became_active":"2026-02-25T08:40:47.016561+0000","last_became_peered":"2026-02-25T08:40:47.016561+0000","last_unstale":"2026-02-25T08:41:20.878402+0000","last_undegraded":"2026-02-25T08:41:20.878402+0000","last_fullsized":"2026-02-25T08:41:20.878402+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.887693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.887693+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.580665+0000","last_change":"2026-02-25T08:41:20.590325+0000","last_active":"2026-02-25T08:41:20.580665+0000","last_peered":"2026-02-25T08:41:20.580665+0000","last_clean":"2026-02-25T08:41:20.580665+0000","last_became_active":"2026-02-25T08:40:47.060230+0000","last_became_peered":"2026-02-25T08:40:47.060230+0000","last_unstale":"2026-02-25T08:41:20.580665+0000","last_undegraded":"2026-02-25T08:41:20.580665+0000","last_fullsized":"2026-02-25T08:41:20.580665+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.590266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.590266+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.579442+0000","last_change":"2026-02-25T08:41:20.593126+0000","last_active":"2026-02-25T08:41:20.579442+0000","last_peered":"2026-02-25T08:41:20.579442+0000","last_clean":"2026-02-25T08:41:20.579442+0000","last_became_active":"2026-02-25T08:40:47.014376+0000","last_became_peered":"2026-02-25T08:40:47.014376+0000","last_unstale":"2026-02-25T08:41:20.579442+0000","last_undegraded":"2026-02-25T08:41:20.579442+0000","last_fullsized":"2026-02-25T08:41:20.579442+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2026-02-25T08:41:20.593072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.593072+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":19,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.281561+0000","last_change":"2026-02-25T08:41:20.291090+0000","last_active":"2026-02-25T08:41:20.281561+0000","last_peered":"2026-02-25T08:41:20.281561+0000","last_clean":"2026-02-25T08:41:20.281561+0000","last_became_active":"2026-02-25T08:40:47.017912+0000","last_became_peered":"2026-02-25T08:40:47.017912+0000","last_unstale":"2026-02-25T08:41:20.281561+0000","last_undegraded":"2026-02-25T08:41:20.281561+0000","last_fullsized":"2026-02-25T08:41:20.281561+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2026-02-25T08:41:20.291018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.291018+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.877972+0000","last_change":"2026-02-25T08:41:20.887056+0000","last_active":"2026-02-25T08:41:20.877972+0000","last_peered":"2026-02-25T08:41:20.877972+0000","last_clean":"2026-02-25T08:41:20.877972+0000","last_became_active":"2026-02-25T08:40:47.003694+0000","last_became_peered":"2026-02-25T08:40:47.003694+0000","last_unstale":"2026-02-25T08:41:20.877972+0000","last_undegraded":"2026-02-25T08:41:20.877972+0000","last_fullsized":"2026-02-25T08:41:20.877972+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.886983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.886983+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-25T08:41:20.578951+0000","last_change":"2026-02-25T08:41:20.587533+0000","last_active":"2026-02-25T08:41:20.578951+0000","last_peered":"2026-02-25T08:41:20.578951+0000","last_clean":"2026-02-25T08:41:20.578951+0000","last_became_active":"2026-02-25T08:40:47.046108+0000","last_became_peered":"2026-02-25T08:40:47.046108+0000","last_unstale":"2026-02-25T08:41:20.578951+0000","last_undegraded":"2026-02-25T08:41:20.578951+0000","last_fullsized":"2026-02-25T08:41:20.578951+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.587450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.587450+0000","last_clean_scrub_stamp":"2026-02-25T08:40:45.957859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160657534976,"internally_reserved":0,"allocated":1085440,"data_stored":1085440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297420,"statfs":{"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":376,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297352,"statfs":{"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53552488448,"internally_reserved":0,"allocated":385024,"data_stored":385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53552558080,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:41.291 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-25T08:41:41.366 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:41.361+0000 7f620cc06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:41.369 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-25T08:41:41.367+0000 7f620cc06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:41.494 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T08:41:41.494 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T08:41:41.495 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T08:41:41.495 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:41.546 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T08:41:41.546 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T08:41:41.546 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:41.608 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T08:41:41.608 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T08:41:41.608 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:41.670 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T08:41:41.670 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T08:41:41.670 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T08:41:41.670 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:41.704 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T08:41:41.704 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T08:41:41.705 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T08:41:41.705 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:41.754 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T08:41:41.754 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-25T08:41:41.754 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:41.804 INFO:tasks.ceph.mon.b:Stopped 2026-02-25T08:41:41.805 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-25T08:41:41.805 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:41.854 INFO:tasks.ceph.mon.c:Stopped 2026-02-25T08:41:41.854 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T08:41:41.855 DEBUG:teuthology.orchestra.run.trial134:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v 'but it is still running' | grep -E -v 'overall HEALTH_' | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2026-02-25T08:41:41.920 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:41:41.921 DEBUG:teuthology.orchestra.run.trial134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T08:41:41.998 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:41:41.998 DEBUG:teuthology.orchestra.run.trial134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T08:41:42.072 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:41:42.072 DEBUG:teuthology.orchestra.run.trial134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T08:41:42.140 INFO:tasks.ceph:Archiving mon data... 2026-02-25T08:41:42.141 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072/data/mon.a.tgz 2026-02-25T08:41:42.142 DEBUG:teuthology.orchestra.run.trial134:> mktemp 2026-02-25T08:41:42.196 INFO:teuthology.orchestra.run.trial134.stdout:/tmp/tmp.UuClOzvwHz 2026-02-25T08:41:42.197 DEBUG:teuthology.orchestra.run.trial134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.UuClOzvwHz 2026-02-25T08:41:42.279 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0666 /tmp/tmp.UuClOzvwHz 2026-02-25T08:41:42.401 DEBUG:teuthology.orchestra.remote:trial134:/tmp/tmp.UuClOzvwHz is 141KB 2026-02-25T08:41:42.452 DEBUG:teuthology.orchestra.run.trial134:> rm -fr /tmp/tmp.UuClOzvwHz 2026-02-25T08:41:42.468 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072/data/mon.b.tgz 2026-02-25T08:41:42.469 DEBUG:teuthology.orchestra.run.trial134:> mktemp 2026-02-25T08:41:42.525 INFO:teuthology.orchestra.run.trial134.stdout:/tmp/tmp.dH5rlNe9nW 2026-02-25T08:41:42.525 DEBUG:teuthology.orchestra.run.trial134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.dH5rlNe9nW 2026-02-25T08:41:42.610 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0666 /tmp/tmp.dH5rlNe9nW 2026-02-25T08:41:42.691 DEBUG:teuthology.orchestra.remote:trial134:/tmp/tmp.dH5rlNe9nW is 196KB 2026-02-25T08:41:42.741 DEBUG:teuthology.orchestra.run.trial134:> rm -fr /tmp/tmp.dH5rlNe9nW 2026-02-25T08:41:42.758 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072/data/mon.c.tgz 2026-02-25T08:41:42.758 DEBUG:teuthology.orchestra.run.trial134:> mktemp 2026-02-25T08:41:42.812 INFO:teuthology.orchestra.run.trial134.stdout:/tmp/tmp.2I0aSDyqBn 2026-02-25T08:41:42.812 DEBUG:teuthology.orchestra.run.trial134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2I0aSDyqBn 2026-02-25T08:41:42.899 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0666 /tmp/tmp.2I0aSDyqBn 2026-02-25T08:41:42.983 DEBUG:teuthology.orchestra.remote:trial134:/tmp/tmp.2I0aSDyqBn is 196KB 2026-02-25T08:41:43.033 DEBUG:teuthology.orchestra.run.trial134:> rm -fr /tmp/tmp.2I0aSDyqBn 2026-02-25T08:41:43.049 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T08:41:43.049 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:41:43.169 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:41:43.169 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T08:41:43.172 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072/remote/trial134/crash 2026-02-25T08:41:43.173 DEBUG:teuthology.orchestra.run.trial134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:41:43.233 INFO:tasks.ceph:Compressing logs... 2026-02-25T08:41:43.233 DEBUG:teuthology.orchestra.run.trial134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:41:43.314 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42331.log 2026-02-25T08:41:43.314 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42331.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42331.log.gz 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41227.log 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43809.log 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42450.log 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41227.log.gz 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43809.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43809.log.gz 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41933.log 2026-02-25T08:41:43.315 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43856.log 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42450.log.gz 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42815.log 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41933.log.gz 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43856.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43856.log.gz 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42183.log 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41274.log 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42815.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42815.log.gz 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42183.log: gzip -5 --verbose 49.6% -- -- replaced with /var/log/ceph/ceph-client.admin.42183.log.gz /var/log/ceph/ceph-client.admin.42766.log 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41274.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41274.log.gz 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2026-02-25T08:41:43.316 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42766.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42766.log.gz 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41773.log 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43515.log 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43662.log 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41773.log.gz 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43564.log 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43515.log.gz 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42182.log 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43662.log.gz 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43564.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43564.log.gz 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42329.log 2026-02-25T08:41:43.317 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42644.log 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42182.log.gz 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42329.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42329.log.gz 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43613.log 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43760.log 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42644.log.gz 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43613.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43613.log.gz 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40797.log 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43760.log.gz 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.40797.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40797.log.gz 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42717.log 2026-02-25T08:41:43.318 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43466.log 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42717.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42717.log.gz 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43466.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43466.log.gz 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42597.log 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.40987.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39300.log 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42597.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42597.log.gz 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42181.log 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph.tmp-client.admin.39300.log: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39300.log.gz 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41058.log 2026-02-25T08:41:43.319 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42181.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42181.log.gz 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41326.log 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41058.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41058.log.gz 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41326.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41326.log.gz 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42497.log 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-mon.b.log: 94.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43417.log 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42497.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42497.log.gz 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43119.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2026-02-25T08:41:43.320 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42570.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43417.log.gz 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41712.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42080.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42570.log.gz 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41712.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41712.log.gz 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41540.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42080.log.gz 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41982.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42276.log 2026-02-25T08:41:43.321 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41540.log.gz 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43095.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41648.log 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42276.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42276.log.gz 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43319.log 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41648.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41648.log.gz 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43319.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43319.log.gz 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42031.log 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43368.log 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42031.log: 93.3% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42031.log.gz 2026-02-25T08:41:43.322 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43270.log 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41843.log 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43368.log.gz 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43270.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43270.log.gz 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43711.log 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41843.log.gz 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.43711.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43711.log.gz 2026-02-25T08:41:43.323 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41105.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2026-02-25T08:41:43.324 INFO:teuthology.orchestra.run.trial134.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T08:41:43.325 INFO:teuthology.orchestra.run.trial134.stderr: 88.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-25T08:41:43.329 INFO:teuthology.orchestra.run.trial134.stderr: 88.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-25T08:41:43.337 INFO:teuthology.orchestra.run.trial134.stderr: 88.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T08:41:43.408 INFO:teuthology.orchestra.run.trial134.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T08:41:43.411 INFO:teuthology.orchestra.run.trial134.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T08:41:43.424 INFO:teuthology.orchestra.run.trial134.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T08:41:43.426 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-25T08:41:43.426 INFO:teuthology.orchestra.run.trial134.stderr:real 0m0.136s 2026-02-25T08:41:43.426 INFO:teuthology.orchestra.run.trial134.stderr:user 0m0.422s 2026-02-25T08:41:43.427 INFO:teuthology.orchestra.run.trial134.stderr:sys 0m0.060s 2026-02-25T08:41:43.427 INFO:tasks.ceph:Archiving logs... 2026-02-25T08:41:43.427 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072/remote/trial134/log 2026-02-25T08:41:43.429 DEBUG:teuthology.orchestra.run.trial134:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:41:43.701 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:41:43.772 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:41:43.783 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T08:41:43.783 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:41:43.809 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T08:41:43.820 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T08:41:43.820 DEBUG:teuthology.orchestra.run.trial134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:41:43.864 INFO:teuthology.orchestra.run.trial134.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:41:43.880 INFO:teuthology.orchestra.run.trial134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:41:43.881 INFO:teuthology.orchestra.run.trial134.stdout:=============================================================================== 2026-02-25T08:41:43.881 INFO:teuthology.orchestra.run.trial134.stdout:^- ntp1.wiktel.com 1 6 377 4 -1641us[-1641us] +/- 21ms 2026-02-25T08:41:43.881 INFO:teuthology.orchestra.run.trial134.stdout:^- ntp.shastacoe.net 2 8 377 132 +743us[ +743us] +/- 67ms 2026-02-25T08:41:43.881 INFO:teuthology.orchestra.run.trial134.stdout:^- arm1.maxhost.io 2 8 377 133 +176us[ +176us] +/- 54ms 2026-02-25T08:41:43.881 INFO:teuthology.orchestra.run.trial134.stdout:^* pool-138-89-14-60.mad.ea> 1 8 377 257 +65us[ +101us] +/- 2934us 2026-02-25T08:41:43.882 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T08:41:43.894 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T08:41:43.894 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T08:41:43.911 DEBUG:teuthology.orchestra.run.trial134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:41:43.977 DEBUG:teuthology.orchestra.run.trial134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:41:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:41:44.003 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T08:41:44.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T08:41:44.036 INFO:teuthology.task.internal:Duration was 2163.753570 seconds 2026-02-25T08:41:44.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T08:41:44.054 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T08:41:44.054 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:41:44.091 INFO:teuthology.orchestra.run.trial134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:41:44.530 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T08:41:44.530 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial134.front.sepia.ceph.com 2026-02-25T08:41:44.531 DEBUG:teuthology.orchestra.run.trial134:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:41:44.591 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T08:41:44.591 DEBUG:teuthology.orchestra.run.trial134:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:41:44.693 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T08:41:44.693 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:41:44.716 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:41:44.718 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:41:44.718 INFO:teuthology.orchestra.run.trial134.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:41:44.718 INFO:teuthology.orchestra.run.trial134.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:41:44.718 INFO:teuthology.orchestra.run.trial134.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:41:44.727 INFO:teuthology.orchestra.run.trial134.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:41:44.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T08:41:44.747 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T08:41:44.747 DEBUG:teuthology.orchestra.run.trial134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:41:44.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T08:41:44.805 DEBUG:teuthology.orchestra.run.trial134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:41:44.856 INFO:teuthology.orchestra.run.trial134.stdout:kernel.core_pattern = core 2026-02-25T08:41:44.868 DEBUG:teuthology.orchestra.run.trial134:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:41:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:41:44.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T08:41:44.939 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T08:41:44.940 DEBUG:teuthology.misc:Transferring archived files from trial134:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69072/remote/trial134 2026-02-25T08:41:44.940 DEBUG:teuthology.orchestra.run.trial134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:41:44.996 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T08:41:44.996 DEBUG:teuthology.orchestra.run.trial134:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:41:45.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T08:41:45.056 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T08:41:45.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T08:41:45.070 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T08:41:45.070 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:41:45.100 INFO:teuthology.orchestra.run.trial134.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:41 /home/ubuntu/cephtest 2026-02-25T08:41:45.102 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T08:41:45.120 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T08:41:45.148 INFO:teuthology.run:Summary data: description: crimson-rados/singleton/{0-crimson_install all/osd-dump-metrics crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/seastore/segmented$/{crimson_seastore_segmented} rados} duration: 2163.7535700798035 flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com success: true 2026-02-25T08:41:45.148 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:41:45.241 INFO:teuthology.run:pass