2026-02-26T03:06:04.337 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T03:06:04.355 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:06:04.438 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71174 branch: main description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} tasks/rbd_python_api_tests} email: null first_in_suite: false flavor: debug job_id: '71174' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - slow request sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: extra_system_packages: - python3-pytest flavor: debug sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5117 sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 targets: trial095.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB7arm2FG3dyz5mi5X7eO7jbnddcCgGahAOiLWP3mLYxQspKKPF2KHv4ooJ93kooQKJVq8NpM3ctPKSF4gExYS0= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 - workunit: clients: client.0: - rbd/test_librbd_python.sh env: RBD_FEATURES: '61' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_02:22:56 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T03:06:04.438 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa; will attempt to use it 2026-02-26T03:06:04.439 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa/tasks 2026-02-26T03:06:04.439 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T03:06:04.441 INFO:teuthology.task.internal:Checking packages... 2026-02-26T03:06:04.441 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'abffb617683951511fccf3c3cab6e219f2ec83a8' 2026-02-26T03:06:04.441 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T03:06:04.441 INFO:teuthology.packaging:ref: None 2026-02-26T03:06:04.441 INFO:teuthology.packaging:tag: None 2026-02-26T03:06:04.441 INFO:teuthology.packaging:branch: main 2026-02-26T03:06:04.441 INFO:teuthology.packaging:sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:06:04.441 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-26T03:06:04.572 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5593.g63656871 2026-02-26T03:06:04.574 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T03:06:04.586 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T03:06:04.586 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T03:06:04.600 INFO:teuthology.task.internal:Saving configuration 2026-02-26T03:06:04.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T03:06:04.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T03:06:04.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial095.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71174', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 03:02:40.856592', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB7arm2FG3dyz5mi5X7eO7jbnddcCgGahAOiLWP3mLYxQspKKPF2KHv4ooJ93kooQKJVq8NpM3ctPKSF4gExYS0='} 2026-02-26T03:06:04.706 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T03:06:04.719 INFO:teuthology.task.internal:roles: ubuntu@trial095.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T03:06:04.719 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T03:06:04.807 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7b55bde24680>, signals=[15]) 2026-02-26T03:06:04.807 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T03:06:04.830 INFO:teuthology.task.internal:Opening connections... 2026-02-26T03:06:04.830 DEBUG:teuthology.task.internal:connecting to ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:06:04.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:06:04.902 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T03:06:04.911 DEBUG:teuthology.orchestra.run.trial095:> uname -m 2026-02-26T03:06:04.977 INFO:teuthology.orchestra.run.trial095.stdout:x86_64 2026-02-26T03:06:04.977 DEBUG:teuthology.orchestra.run.trial095:> cat /etc/os-release 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:NAME="CentOS Stream" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:VERSION="9" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:ID="centos" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:ID_LIKE="rhel fedora" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:VERSION_ID="9" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:PLATFORM_ID="platform:el9" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:ANSI_COLOR="0;31" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:LOGO="fedora-logo-icon" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:HOME_URL="https://centos.org/" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T03:06:05.033 INFO:teuthology.orchestra.run.trial095.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T03:06:05.034 INFO:teuthology.lock.ops:Updating trial095.front.sepia.ceph.com on lock server 2026-02-26T03:06:05.118 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T03:06:05.131 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T03:06:05.142 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T03:06:05.142 DEBUG:teuthology.orchestra.run.trial095:> test '!' -e /home/ubuntu/cephtest 2026-02-26T03:06:05.158 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T03:06:05.169 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T03:06:05.169 DEBUG:teuthology.orchestra.run.trial095:> test -z $(ls -A /var/lib/ceph) 2026-02-26T03:06:05.214 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T03:06:05.324 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T03:06:05.345 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T03:06:05.345 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T03:06:05.345 DEBUG:teuthology.orchestra.run.trial095:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T03:06:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:06:05.360 DEBUG:teuthology.orchestra.run.trial095:> uname -r 2026-02-26T03:06:05.416 INFO:teuthology.orchestra.run.trial095.stdout:5.14.0-681.el9.x86_64 2026-02-26T03:06:05.416 INFO:teuthology.task.kernel:Running kernel on trial095: 5.14.0-681.el9.x86_64 2026-02-26T03:06:05.416 DEBUG:teuthology.orchestra.run.trial095:> sudo yum install -y kernel 2026-02-26T03:06:06.585 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-26T03:06:35.754 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - AppStream 964 kB/s | 27 MB 00:28 2026-02-26T03:06:39.431 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 8.0 MB 00:01 2026-02-26T03:06:40.314 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - Extras packages 138 kB/s | 20 kB 00:00 2026-02-26T03:06:42.185 INFO:teuthology.orchestra.run.trial095.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2026-02-26T03:06:45.415 INFO:teuthology.orchestra.run.trial095.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-26T03:06:46.444 INFO:teuthology.orchestra.run.trial095.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T03:06:46.444 INFO:teuthology.orchestra.run.trial095.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T03:06:46.464 INFO:teuthology.orchestra.run.trial095.stdout:Dependencies resolved. 2026-02-26T03:06:46.465 INFO:teuthology.orchestra.run.trial095.stdout:================================================================================ 2026-02-26T03:06:46.465 INFO:teuthology.orchestra.run.trial095.stdout: Package Architecture Version Repository Size 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout:================================================================================ 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout:Installing: 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout:Installing dependencies: 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout:Transaction Summary 2026-02-26T03:06:46.466 INFO:teuthology.orchestra.run.trial095.stdout:================================================================================ 2026-02-26T03:06:46.467 INFO:teuthology.orchestra.run.trial095.stdout:Install 4 Packages 2026-02-26T03:06:46.467 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:06:46.467 INFO:teuthology.orchestra.run.trial095.stdout:Total download size: 89 M 2026-02-26T03:06:46.467 INFO:teuthology.orchestra.run.trial095.stdout:Installed size: 135 M 2026-02-26T03:06:46.467 INFO:teuthology.orchestra.run.trial095.stdout:Downloading Packages: 2026-02-26T03:06:46.557 INFO:teuthology.orchestra.run.trial095.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 30 MB/s | 901 kB 00:00 2026-02-26T03:06:46.684 INFO:teuthology.orchestra.run.trial095.stdout:(2/4): kernel-modules-core-5.14.0-686.el9.x86_6 245 MB/s | 31 MB 00:00 2026-02-26T03:06:46.746 INFO:teuthology.orchestra.run.trial095.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 182 MB/s | 40 MB 00:00 2026-02-26T03:06:46.870 INFO:teuthology.orchestra.run.trial095.stdout:(4/4): kernel-core-5.14.0-686.el9.x86_64.rpm 50 MB/s | 17 MB 00:00 2026-02-26T03:06:46.870 INFO:teuthology.orchestra.run.trial095.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:06:46.870 INFO:teuthology.orchestra.run.trial095.stdout:Total 220 MB/s | 89 MB 00:00 2026-02-26T03:06:47.060 INFO:teuthology.orchestra.run.trial095.stdout:Running transaction check 2026-02-26T03:06:47.262 INFO:teuthology.orchestra.run.trial095.stdout:Transaction check succeeded. 2026-02-26T03:06:47.263 INFO:teuthology.orchestra.run.trial095.stdout:Running transaction test 2026-02-26T03:06:47.648 INFO:teuthology.orchestra.run.trial095.stdout:Transaction test succeeded. 2026-02-26T03:06:47.648 INFO:teuthology.orchestra.run.trial095.stdout:Running transaction 2026-02-26T03:06:48.763 INFO:teuthology.orchestra.run.trial095.stdout: Preparing : 1/1 2026-02-26T03:06:49.055 INFO:teuthology.orchestra.run.trial095.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:06:49.077 INFO:teuthology.orchestra.run.trial095.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:06:51.640 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:06:51.673 INFO:teuthology.orchestra.run.trial095.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:06:54.916 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:06:54.923 INFO:teuthology.orchestra.run.trial095.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:06:58.040 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:07:15.522 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:07:15.545 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:07:15.733 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:07:15.733 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:07:15.733 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:07:15.733 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout:Installed: 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:07:15.790 INFO:teuthology.orchestra.run.trial095.stdout:Complete! 2026-02-26T03:07:15.852 DEBUG:teuthology.orchestra.run.trial095:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:07:15.881 INFO:teuthology.orchestra.run.trial095.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:07:15.882 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:07:15.882 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T03:07:15.882 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T03:07:15.882 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T03:07:15.882 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T03:07:15.882 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial095.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T03:07:15.883 DEBUG:teuthology.orchestra.run.trial095:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:07:15.948 INFO:teuthology.orchestra.run.trial095.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:07:15.948 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:07:15.949 DEBUG:teuthology.orchestra.run.trial095:> sudo rpm -qi grub2-tools 2026-02-26T03:07:16.017 INFO:teuthology.orchestra.run.trial095.stdout:Name : grub2-tools 2026-02-26T03:07:16.017 INFO:teuthology.orchestra.run.trial095.stdout:Epoch : 1 2026-02-26T03:07:16.017 INFO:teuthology.orchestra.run.trial095.stdout:Version : 2.06 2026-02-26T03:07:16.017 INFO:teuthology.orchestra.run.trial095.stdout:Release : 120.el9 2026-02-26T03:07:16.017 INFO:teuthology.orchestra.run.trial095.stdout:Architecture: x86_64 2026-02-26T03:07:16.017 INFO:teuthology.orchestra.run.trial095.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Group : Unspecified 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Size : 9108611 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:License : GPLv3+ 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Packager : builder@centos.org 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Vendor : CentOS 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Summary : Support tools for GRUB. 2026-02-26T03:07:16.018 INFO:teuthology.orchestra.run.trial095.stdout:Description : 2026-02-26T03:07:16.019 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:07:16.019 INFO:teuthology.orchestra.run.trial095.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T03:07:16.019 INFO:teuthology.orchestra.run.trial095.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T03:07:16.019 INFO:teuthology.orchestra.run.trial095.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T03:07:16.019 INFO:teuthology.orchestra.run.trial095.stdout:hardware devices. 2026-02-26T03:07:16.019 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:07:16.019 INFO:teuthology.orchestra.run.trial095.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T03:07:16.020 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T03:07:16.020 INFO:teuthology.task.kernel:Updating grub on trial095 to boot 5.14.0-686.el9.x86_64 2026-02-26T03:07:16.020 DEBUG:teuthology.orchestra.run.trial095:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T03:07:16.086 INFO:teuthology.orchestra.run.trial095.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T03:07:16.087 DEBUG:teuthology.orchestra.run.trial095:> sudo command -v grubby && echo yes || echo no 2026-02-26T03:07:16.150 INFO:teuthology.orchestra.run.trial095.stdout:/sbin/grubby 2026-02-26T03:07:16.152 INFO:teuthology.orchestra.run.trial095.stdout:yes 2026-02-26T03:07:16.152 DEBUG:teuthology.orchestra.run.trial095:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T03:07:16.218 INFO:teuthology.orchestra.run.trial095.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T03:07:16.218 DEBUG:teuthology.orchestra.run.trial095:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T03:07:16.291 DEBUG:teuthology.orchestra.run.trial095:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T03:07:16.500 INFO:teuthology.orchestra.run.trial095.stderr:Generating grub configuration file ... 2026-02-26T03:07:16.566 INFO:teuthology.orchestra.run.trial095.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T03:07:16.572 INFO:teuthology.orchestra.run.trial095.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T03:07:16.823 INFO:teuthology.orchestra.run.trial095.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T03:07:16.829 INFO:teuthology.orchestra.run.trial095.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T03:07:16.863 INFO:teuthology.orchestra.run.trial095.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T03:07:16.869 INFO:teuthology.orchestra.run.trial095.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T03:07:16.901 INFO:teuthology.orchestra.run.trial095.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T03:07:16.907 INFO:teuthology.orchestra.run.trial095.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T03:07:16.932 INFO:teuthology.orchestra.run.trial095.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T03:07:16.938 INFO:teuthology.orchestra.run.trial095.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T03:07:17.024 INFO:teuthology.orchestra.run.trial095.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T03:07:17.033 INFO:teuthology.orchestra.run.trial095.stderr:done 2026-02-26T03:07:17.035 DEBUG:teuthology.orchestra.run.trial095:> test -d /sys/firmware/efi 2026-02-26T03:07:17.052 DEBUG:teuthology.orchestra.run.trial095:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T03:07:17.119 INFO:teuthology.orchestra.run.trial095.stdout:/boot/efi/EFI/BOOT 2026-02-26T03:07:17.119 INFO:teuthology.orchestra.run.trial095.stdout:/boot/efi/EFI/centos 2026-02-26T03:07:17.121 DEBUG:teuthology.orchestra.run.trial095:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T03:07:17.186 INFO:teuthology.orchestra.run.trial095.stdout:no 2026-02-26T03:07:17.186 DEBUG:teuthology.orchestra.run.trial095:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T03:07:17.249 INFO:teuthology.orchestra.run.trial095.stdout:yes 2026-02-26T03:07:17.249 DEBUG:teuthology.orchestra.run.trial095:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T03:07:17.312 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial095 2026-02-26T03:07:17.313 DEBUG:teuthology.orchestra.run.trial095:> sudo bash -c ' 2026-02-26T03:07:17.313 DEBUG:teuthology.orchestra.run.trial095:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T03:07:17.313 DEBUG:teuthology.orchestra.run.trial095:> ' 2026-02-26T03:07:47.355 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T03:07:47.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial095.front.sepia.ceph.com' 2026-02-26T03:07:47.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:08:47.408 DEBUG:teuthology.orchestra.remote:timed out 2026-02-26T03:08:56.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial095.front.sepia.ceph.com' 2026-02-26T03:08:56.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:08:56.541 DEBUG:teuthology.orchestra.run.trial095:> true 2026-02-26T03:08:56.683 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial095.front.sepia.ceph.com' 2026-02-26T03:08:56.683 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T03:08:56.683 DEBUG:teuthology.orchestra.run.trial095:> uname -r 2026-02-26T03:08:56.739 INFO:teuthology.orchestra.run.trial095.stdout:5.14.0-686.el9.x86_64 2026-02-26T03:08:56.740 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T03:08:56.740 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T03:08:56.740 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T03:08:57.740 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T03:08:57.740 DEBUG:teuthology.orchestra.run.trial095:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T03:08:57.770 INFO:teuthology.orchestra.run.trial095.stdout:ttyS1 2026-02-26T03:08:57.794 DEBUG:teuthology.parallel:result is None 2026-02-26T03:08:57.795 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T03:08:57.807 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T03:08:57.808 DEBUG:teuthology.orchestra.run.trial095:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T03:08:57.851 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T03:08:57.863 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T03:08:57.878 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T03:08:57.878 DEBUG:teuthology.orchestra.run.trial095:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T03:08:57.909 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T03:08:57.921 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T03:08:57.921 DEBUG:teuthology.orchestra.run.trial095:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T03:08:57.972 INFO:teuthology.orchestra.run.trial095.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:08:57.980 INFO:teuthology.orchestra.run.trial095.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:08:57.982 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T03:08:57.993 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T03:08:57.994 DEBUG:teuthology.orchestra.run.trial095:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T03:08:58.047 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T03:08:58.067 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T03:08:58.067 DEBUG:teuthology.orchestra.run.trial095:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T03:08:58.103 DEBUG:teuthology.orchestra.run.trial095:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:08:58.265 DEBUG:teuthology.orchestra.run.trial095:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:08:58.291 DEBUG:teuthology.orchestra.run.trial095:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:08:58.450 DEBUG:teuthology.orchestra.run.trial095:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:08:58.475 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:08:58.475 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T03:08:58.538 DEBUG:teuthology.orchestra.run.trial095:> sudo service rsyslog restart 2026-02-26T03:08:58.605 INFO:teuthology.orchestra.run.trial095.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:08:59.004 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T03:08:59.016 INFO:teuthology.task.internal:Starting timer... 2026-02-26T03:08:59.017 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T03:08:59.053 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T03:08:59.068 DEBUG:teuthology.orchestra.run.trial095:> sudo service auditd rotate 2026-02-26T03:08:59.096 INFO:teuthology.orchestra.run.trial095.stdout:Rotating logs: 2026-02-26T03:08:59.097 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T03:08:59.097 DEBUG:teuthology.orchestra.run.trial095:> /usr/sbin/getenforce 2026-02-26T03:08:59.153 INFO:teuthology.orchestra.run.trial095.stdout:Permissive 2026-02-26T03:08:59.153 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial095.front.sepia.ceph.com': 'permissive'} 2026-02-26T03:08:59.154 DEBUG:teuthology.orchestra.run.trial095:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:08:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:08:59.217 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T03:08:59.218 DEBUG:teuthology.orchestra.run.trial095:> sudo /usr/sbin/setenforce permissive 2026-02-26T03:08:59.280 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T03:08:59.297 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T03:08:59.312 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T03:08:59.314 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial095.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T03:11:46.972 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial095.front.sepia.ceph.com')] 2026-02-26T03:11:46.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial095.front.sepia.ceph.com' 2026-02-26T03:11:46.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:11:47.048 DEBUG:teuthology.orchestra.run.trial095:> true 2026-02-26T03:11:47.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial095.front.sepia.ceph.com' 2026-02-26T03:11:47.125 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T03:11:47.142 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T03:11:47.142 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T03:11:47.143 DEBUG:teuthology.orchestra.run.trial095:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:11:47.192 INFO:teuthology.orchestra.run.trial095.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T03:11:47.204 INFO:teuthology.orchestra.run.trial095.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T03:11:47.225 INFO:teuthology.orchestra.run.trial095.stderr:sudo: ntpd: command not found 2026-02-26T03:11:47.235 INFO:teuthology.orchestra.run.trial095.stdout:506 Cannot talk to daemon 2026-02-26T03:11:47.248 INFO:teuthology.orchestra.run.trial095.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T03:11:47.259 INFO:teuthology.orchestra.run.trial095.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T03:11:47.311 INFO:teuthology.orchestra.run.trial095.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:11:47.489 INFO:teuthology.orchestra.run.trial095.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:11:47.489 INFO:teuthology.orchestra.run.trial095.stdout:=============================================================================== 2026-02-26T03:11:47.489 INFO:teuthology.orchestra.run.trial095.stdout:^? 23.186.168.132 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:11:47.490 INFO:teuthology.orchestra.run.trial095.stdout:^? ntp2.zeus.frumentum.media 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:11:47.490 INFO:teuthology.orchestra.run.trial095.stdout:^? ntp6.kernfusion.at 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:11:47.490 INFO:teuthology.orchestra.run.trial095.stdout:^? chi2.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:11:47.491 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:11:47.506 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:11:47.506 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:11:47.506 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:11:47.507 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:11:47.517 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:11:47.518 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-26T03:11:47.518 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:11:47.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:11:47.678 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:11:47.678 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:11:48.056 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:11:48.056 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:11:48.056 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:11:48.083 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo, python3-pytest on remote rpm x86_64 2026-02-26T03:11:48.083 DEBUG:teuthology.orchestra.run.trial095:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:11:48.148 DEBUG:teuthology.orchestra.run.trial095:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:11:48.222 DEBUG:teuthology.orchestra.run.trial095:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:11:48.285 INFO:teuthology.orchestra.run.trial095.stdout:check_obsoletes = 1 2026-02-26T03:11:48.286 DEBUG:teuthology.orchestra.run.trial095:> sudo yum clean all 2026-02-26T03:11:48.433 INFO:teuthology.orchestra.run.trial095.stdout:41 files removed 2026-02-26T03:11:48.447 DEBUG:teuthology.orchestra.run.trial095:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-osd-crimson ceph-osd-crimson-debuginfo python3-pytest 2026-02-26T03:11:49.270 INFO:teuthology.orchestra.run.trial095.stdout:ceph packages for x86_64 140 kB/s | 94 kB 00:00 2026-02-26T03:11:49.914 INFO:teuthology.orchestra.run.trial095.stdout:ceph noarch packages 40 kB/s | 24 kB 00:00 2026-02-26T03:11:50.458 INFO:teuthology.orchestra.run.trial095.stdout:ceph source packages 4.0 kB/s | 2.1 kB 00:00 2026-02-26T03:11:52.194 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.9 MB 00:01 2026-02-26T03:11:55.264 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-26T03:11:58.899 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 8.0 MB 00:01 2026-02-26T03:11:59.851 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 20 kB 00:00 2026-02-26T03:12:00.467 INFO:teuthology.orchestra.run.trial095.stdout:Extra Packages for Enterprise Linux 37 MB/s | 20 MB 00:00 2026-02-26T03:12:03.695 INFO:teuthology.orchestra.run.trial095.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-26T03:12:04.922 INFO:teuthology.orchestra.run.trial095.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2026-02-26T03:12:04.922 INFO:teuthology.orchestra.run.trial095.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2026-02-26T03:12:04.950 INFO:teuthology.orchestra.run.trial095.stdout:Dependencies resolved. 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout:======================================================================================= 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: Package Arch Version Repository Size 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout:======================================================================================= 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout:Installing: 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph x86_64 2:20.3.0-5312.gabffb617.el9 ceph 6.5 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-base x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.1 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 914 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.gabffb617.el9 ceph 149 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr x86_64 2:20.3.0-5312.gabffb617.el9 ceph 937 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 184 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 51 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 7.4 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 50 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 163 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.gabffb617.el9 ceph 15 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.gabffb617.el9 ceph 429 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-radosgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 11 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-test x86_64 2:20.3.0-5312.gabffb617.el9 ceph 126 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: ceph-volume noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 301 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 1.0 M 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 859 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: librados-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 124 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: python3-cephfs x86_64 2:20.3.0-5312.gabffb617.el9 ceph 203 k 2026-02-26T03:12:04.955 INFO:teuthology.orchestra.run.trial095.stdout: python3-pytest noarch 6.2.2-7.el9 appstream 519 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: python3-rados x86_64 2:20.3.0-5312.gabffb617.el9 ceph 324 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: python3-rbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 305 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: python3-rgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 100 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: rbd-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: rbd-mirror x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.6 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: rbd-nbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 166 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout:Upgrading: 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: librados2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 3.3 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: librbd1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.7 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout:Installing dependencies: 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 21 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 57 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mds x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.0 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 297 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 4.9 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 17 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ceph-selinux x86_64 2:20.3.0-5312.gabffb617.el9 ceph 25 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 26 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libcephsqlite x86_64 2:20.3.0-5312.gabffb617.el9 ceph 153 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T03:12:04.956 INFO:teuthology.orchestra.run.trial095.stdout: libradosstriper1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 263 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: librgw2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.6 M 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 45 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 185 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-iniconfig noarch 1.1.1-7.el9 appstream 17 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:12:04.957 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-pluggy noarch 0.13.1-7.el9 appstream 41 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-py noarch 1.10.0-6.el9 appstream 477 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T03:12:04.958 INFO:teuthology.orchestra.run.trial095.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout:Installing weak dependencies: 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: ceph-debugsource x86_64 2:20.3.0-5312.gabffb617.el9 ceph 12 M 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 22 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout:Transaction Summary 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout:======================================================================================= 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout:Install 156 Packages 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout:Upgrade 2 Packages 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout:Total download size: 863 M 2026-02-26T03:12:04.959 INFO:teuthology.orchestra.run.trial095.stdout:Downloading Packages: 2026-02-26T03:12:07.006 INFO:teuthology.orchestra.run.trial095.stdout:(1/158): ceph-20.3.0-5312.gabffb617.el9.x86_64. 22 kB/s | 6.5 kB 00:00 2026-02-26T03:12:08.967 INFO:teuthology.orchestra.run.trial095.stdout:(2/158): ceph-base-20.3.0-5312.gabffb617.el9.x8 2.3 MB/s | 5.1 MB 00:02 2026-02-26T03:12:09.372 INFO:teuthology.orchestra.run.trial095.stdout:(3/158): ceph-fuse-20.3.0-5312.gabffb617.el9.x8 2.2 MB/s | 914 kB 00:00 2026-02-26T03:12:09.438 INFO:teuthology.orchestra.run.trial095.stdout:(4/158): ceph-immutable-object-cache-20.3.0-531 2.2 MB/s | 149 kB 00:00 2026-02-26T03:12:10.518 INFO:teuthology.orchestra.run.trial095.stdout:(5/158): ceph-mds-20.3.0-5312.gabffb617.el9.x86 1.9 MB/s | 2.0 MB 00:01 2026-02-26T03:12:11.033 INFO:teuthology.orchestra.run.trial095.stdout:(6/158): ceph-debugsource-20.3.0-5312.gabffb617 3.0 MB/s | 12 MB 00:04 2026-02-26T03:12:11.291 INFO:teuthology.orchestra.run.trial095.stdout:(7/158): ceph-mgr-20.3.0-5312.gabffb617.el9.x86 1.2 MB/s | 937 kB 00:00 2026-02-26T03:12:11.487 INFO:teuthology.orchestra.run.trial095.stdout:(8/158): ceph-osd-20.3.0-5312.gabffb617.el9.x86 833 kB/s | 163 kB 00:00 2026-02-26T03:12:14.370 INFO:teuthology.orchestra.run.trial095.stdout:(9/158): ceph-mon-20.3.0-5312.gabffb617.el9.x86 1.5 MB/s | 4.9 MB 00:03 2026-02-26T03:12:19.080 INFO:teuthology.orchestra.run.trial095.stdout:(10/158): ceph-common-20.3.0-5312.gabffb617.el9 1.7 MB/s | 21 MB 00:12 2026-02-26T03:12:24.192 INFO:teuthology.orchestra.run.trial095.stdout:(11/158): ceph-osd-classic-20.3.0-5312.gabffb61 1.2 MB/s | 15 MB 00:12 2026-02-26T03:12:31.513 INFO:teuthology.orchestra.run.trial095.stdout:(12/158): ceph-radosgw-20.3.0-5312.gabffb617.el 1.5 MB/s | 11 MB 00:07 2026-02-26T03:12:31.566 INFO:teuthology.orchestra.run.trial095.stdout:(13/158): ceph-selinux-20.3.0-5312.gabffb617.el 473 kB/s | 25 kB 00:00 2026-02-26T03:13:12.365 INFO:teuthology.orchestra.run.trial095.stdout:(14/158): ceph-osd-crimson-20.3.0-5312.gabffb61 1.5 MB/s | 87 MB 00:57 2026-02-26T03:13:12.432 INFO:teuthology.orchestra.run.trial095.stdout:(15/158): libcephfs-daemon-20.3.0-5312.gabffb61 552 kB/s | 37 kB 00:00 2026-02-26T03:13:12.500 INFO:teuthology.orchestra.run.trial095.stdout:(16/158): libcephfs-devel-20.3.0-5312.gabffb617 551 kB/s | 37 kB 00:00 2026-02-26T03:13:12.569 INFO:teuthology.orchestra.run.trial095.stdout:(17/158): libcephfs-proxy2-20.3.0-5312.gabffb61 373 kB/s | 26 kB 00:00 2026-02-26T03:13:13.118 INFO:teuthology.orchestra.run.trial095.stdout:(18/158): libcephfs2-20.3.0-5312.gabffb617.el9. 1.5 MB/s | 859 kB 00:00 2026-02-26T03:13:13.326 INFO:teuthology.orchestra.run.trial095.stdout:(19/158): libcephsqlite-20.3.0-5312.gabffb617.e 738 kB/s | 153 kB 00:00 2026-02-26T03:13:13.462 INFO:teuthology.orchestra.run.trial095.stdout:(20/158): librados-devel-20.3.0-5312.gabffb617. 902 kB/s | 124 kB 00:00 2026-02-26T03:13:13.675 INFO:teuthology.orchestra.run.trial095.stdout:(21/158): libradosstriper1-20.3.0-5312.gabffb61 1.2 MB/s | 263 kB 00:00 2026-02-26T03:13:18.774 INFO:teuthology.orchestra.run.trial095.stdout:(22/158): librgw2-20.3.0-5312.gabffb617.el9.x86 1.1 MB/s | 5.6 MB 00:05 2026-02-26T03:13:18.846 INFO:teuthology.orchestra.run.trial095.stdout:(23/158): python3-ceph-argparse-20.3.0-5312.gab 633 kB/s | 45 kB 00:00 2026-02-26T03:13:19.056 INFO:teuthology.orchestra.run.trial095.stdout:(24/158): python3-ceph-common-20.3.0-5312.gabff 878 kB/s | 185 kB 00:00 2026-02-26T03:13:19.257 INFO:teuthology.orchestra.run.trial095.stdout:(25/158): python3-cephfs-20.3.0-5312.gabffb617. 1.0 MB/s | 203 kB 00:00 2026-02-26T03:13:19.656 INFO:teuthology.orchestra.run.trial095.stdout:(26/158): python3-rados-20.3.0-5312.gabffb617.e 814 kB/s | 324 kB 00:00 2026-02-26T03:13:20.051 INFO:teuthology.orchestra.run.trial095.stdout:(27/158): python3-rbd-20.3.0-5312.gabffb617.el9 772 kB/s | 305 kB 00:00 2026-02-26T03:13:20.189 INFO:teuthology.orchestra.run.trial095.stdout:(28/158): python3-rgw-20.3.0-5312.gabffb617.el9 721 kB/s | 100 kB 00:00 2026-02-26T03:13:20.332 INFO:teuthology.orchestra.run.trial095.stdout:(29/158): rbd-fuse-20.3.0-5312.gabffb617.el9.x8 610 kB/s | 87 kB 00:00 2026-02-26T03:13:22.796 INFO:teuthology.orchestra.run.trial095.stdout:(30/158): rbd-mirror-20.3.0-5312.gabffb617.el9. 1.1 MB/s | 2.6 MB 00:02 2026-02-26T03:13:22.944 INFO:teuthology.orchestra.run.trial095.stdout:(31/158): rbd-nbd-20.3.0-5312.gabffb617.el9.x86 1.1 MB/s | 166 kB 00:00 2026-02-26T03:13:23.021 INFO:teuthology.orchestra.run.trial095.stdout:(32/158): ceph-grafana-dashboards-20.3.0-5312.g 752 kB/s | 57 kB 00:00 2026-02-26T03:13:23.226 INFO:teuthology.orchestra.run.trial095.stdout:(33/158): ceph-mgr-cephadm-20.3.0-5312.gabffb61 897 kB/s | 184 kB 00:00 2026-02-26T03:14:00.423 INFO:teuthology.orchestra.run.trial095.stdout:(34/158): ceph-test-20.3.0-5312.gabffb617.el9.x 1.4 MB/s | 126 MB 01:28 2026-02-26T03:14:04.728 INFO:teuthology.orchestra.run.trial095.stdout:(35/158): ceph-mgr-diskprediction-local-20.3.0- 1.7 MB/s | 7.4 MB 00:04 2026-02-26T03:14:04.797 INFO:teuthology.orchestra.run.trial095.stdout:(36/158): ceph-mgr-k8sevents-20.3.0-5312.gabffb 322 kB/s | 22 kB 00:00 2026-02-26T03:14:05.006 INFO:teuthology.orchestra.run.trial095.stdout:(37/158): ceph-mgr-modules-core-20.3.0-5312.gab 1.4 MB/s | 297 kB 00:00 2026-02-26T03:14:05.071 INFO:teuthology.orchestra.run.trial095.stdout:(38/158): ceph-mgr-rook-20.3.0-5312.gabffb617.e 778 kB/s | 50 kB 00:00 2026-02-26T03:14:05.135 INFO:teuthology.orchestra.run.trial095.stdout:(39/158): ceph-prometheus-alerts-20.3.0-5312.ga 269 kB/s | 17 kB 00:00 2026-02-26T03:14:05.340 INFO:teuthology.orchestra.run.trial095.stdout:(40/158): ceph-volume-20.3.0-5312.gabffb617.el9 1.4 MB/s | 301 kB 00:00 2026-02-26T03:14:05.972 INFO:teuthology.orchestra.run.trial095.stdout:(41/158): cephadm-20.3.0-5312.gabffb617.el9.noa 1.6 MB/s | 1.0 MB 00:00 2026-02-26T03:14:06.110 INFO:teuthology.orchestra.run.trial095.stdout:(42/158): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.5 MB/s | 351 kB 00:00 2026-02-26T03:14:06.193 INFO:teuthology.orchestra.run.trial095.stdout:(43/158): hwloc-libs-2.4.1-6.el9.x86_64.rpm 25 MB/s | 2.1 MB 00:00 2026-02-26T03:14:06.214 INFO:teuthology.orchestra.run.trial095.stdout:(44/158): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.9 MB/s | 40 kB 00:00 2026-02-26T03:14:06.234 INFO:teuthology.orchestra.run.trial095.stdout:(45/158): libatomic-11.5.0-14.el9.x86_64.rpm 1.1 MB/s | 23 kB 00:00 2026-02-26T03:14:06.256 INFO:teuthology.orchestra.run.trial095.stdout:(46/158): libconfig-1.7.2-9.el9.x86_64.rpm 3.3 MB/s | 72 kB 00:00 2026-02-26T03:14:06.280 INFO:teuthology.orchestra.run.trial095.stdout:(47/158): libgfortran-11.5.0-14.el9.x86_64.rpm 32 MB/s | 794 kB 00:00 2026-02-26T03:14:06.303 INFO:teuthology.orchestra.run.trial095.stdout:(48/158): libquadmath-11.5.0-14.el9.x86_64.rpm 8.1 MB/s | 184 kB 00:00 2026-02-26T03:14:06.324 INFO:teuthology.orchestra.run.trial095.stdout:(49/158): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.3 MB/s | 94 kB 00:00 2026-02-26T03:14:06.344 INFO:teuthology.orchestra.run.trial095.stdout:(50/158): mailcap-2.1.49-5.el9.noarch.rpm 1.6 MB/s | 33 kB 00:00 2026-02-26T03:14:06.376 INFO:teuthology.orchestra.run.trial095.stdout:(51/158): nvme-cli-2.16-1.el9.x86_64.rpm 36 MB/s | 1.2 MB 00:00 2026-02-26T03:14:06.397 INFO:teuthology.orchestra.run.trial095.stdout:(52/158): pciutils-3.7.0-7.el9.x86_64.rpm 4.3 MB/s | 93 kB 00:00 2026-02-26T03:14:06.422 INFO:teuthology.orchestra.run.trial095.stdout:(53/158): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-26T03:14:06.450 INFO:teuthology.orchestra.run.trial095.stdout:(54/158): python3-cryptography-36.0.1-5.el9.x86 44 MB/s | 1.2 MB 00:00 2026-02-26T03:14:06.473 INFO:teuthology.orchestra.run.trial095.stdout:(55/158): python3-ply-3.11-14.el9.noarch.rpm 4.6 MB/s | 106 kB 00:00 2026-02-26T03:14:06.497 INFO:teuthology.orchestra.run.trial095.stdout:(56/158): python3-pycparser-2.20-6.el9.noarch.r 5.7 MB/s | 135 kB 00:00 2026-02-26T03:14:06.520 INFO:teuthology.orchestra.run.trial095.stdout:(57/158): python3-pyparsing-2.4.7-9.el9.noarch. 6.4 MB/s | 150 kB 00:00 2026-02-26T03:14:06.542 INFO:teuthology.orchestra.run.trial095.stdout:(58/158): python3-requests-2.25.1-10.el9.noarch 5.6 MB/s | 126 kB 00:00 2026-02-26T03:14:06.566 INFO:teuthology.orchestra.run.trial095.stdout:(59/158): python3-urllib3-1.26.5-7.el9.noarch.r 8.9 MB/s | 218 kB 00:00 2026-02-26T03:14:06.591 INFO:teuthology.orchestra.run.trial095.stdout:(60/158): unzip-6.0-59.el9.x86_64.rpm 7.0 MB/s | 182 kB 00:00 2026-02-26T03:14:06.615 INFO:teuthology.orchestra.run.trial095.stdout:(61/158): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-26T03:14:07.259 INFO:teuthology.orchestra.run.trial095.stdout:(62/158): boost-program-options-1.75.0-13.el9.x 162 kB/s | 104 kB 00:00 2026-02-26T03:14:07.342 INFO:teuthology.orchestra.run.trial095.stdout:(63/158): flexiblas-3.0.4-9.el9.x86_64.rpm 357 kB/s | 30 kB 00:00 2026-02-26T03:14:07.826 INFO:teuthology.orchestra.run.trial095.stdout:(64/158): flexiblas-netlib-3.0.4-9.el9.x86_64.r 6.2 MB/s | 3.0 MB 00:00 2026-02-26T03:14:07.909 INFO:teuthology.orchestra.run.trial095.stdout:(65/158): flexiblas-openblas-openmp-3.0.4-9.el9 178 kB/s | 15 kB 00:00 2026-02-26T03:14:07.995 INFO:teuthology.orchestra.run.trial095.stdout:(66/158): libasan8-15.2.1-7.el9.x86_64.rpm 5.9 MB/s | 525 kB 00:00 2026-02-26T03:14:08.079 INFO:teuthology.orchestra.run.trial095.stdout:(67/158): libnbd-1.20.3-4.el9.x86_64.rpm 1.9 MB/s | 164 kB 00:00 2026-02-26T03:14:08.163 INFO:teuthology.orchestra.run.trial095.stdout:(68/158): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2026-02-26T03:14:08.247 INFO:teuthology.orchestra.run.trial095.stdout:(69/158): librabbitmq-0.11.0-7.el9.x86_64.rpm 539 kB/s | 45 kB 00:00 2026-02-26T03:14:08.333 INFO:teuthology.orchestra.run.trial095.stdout:(70/158): librdkafka-1.6.1-102.el9.x86_64.rpm 7.5 MB/s | 662 kB 00:00 2026-02-26T03:14:08.418 INFO:teuthology.orchestra.run.trial095.stdout:(71/158): libstoragemgmt-1.10.1-1.el9.x86_64.rp 2.9 MB/s | 246 kB 00:00 2026-02-26T03:14:08.501 INFO:teuthology.orchestra.run.trial095.stdout:(72/158): libubsan-11.5.0-14.el9.x86_64.rpm 2.0 MB/s | 176 kB 00:00 2026-02-26T03:14:08.586 INFO:teuthology.orchestra.run.trial095.stdout:(73/158): libxslt-1.1.34-12.el9.x86_64.rpm 2.7 MB/s | 233 kB 00:00 2026-02-26T03:14:08.641 INFO:teuthology.orchestra.run.trial095.stdout:(74/158): ceph-mgr-dashboard-20.3.0-5312.gabffb 1.1 MB/s | 51 MB 00:45 2026-02-26T03:14:08.670 INFO:teuthology.orchestra.run.trial095.stdout:(75/158): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2026-02-26T03:14:08.754 INFO:teuthology.orchestra.run.trial095.stdout:(76/158): openblas-0.3.29-1.el9.x86_64.rpm 506 kB/s | 42 kB 00:00 2026-02-26T03:14:08.929 INFO:teuthology.orchestra.run.trial095.stdout:(77/158): openblas-openmp-0.3.29-1.el9.x86_64.r 30 MB/s | 5.3 MB 00:00 2026-02-26T03:14:09.017 INFO:teuthology.orchestra.run.trial095.stdout:(78/158): protobuf-3.14.0-17.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2026-02-26T03:14:09.196 INFO:teuthology.orchestra.run.trial095.stdout:(79/158): python3-babel-2.9.1-2.el9.noarch.rpm 34 MB/s | 6.0 MB 00:00 2026-02-26T03:14:09.206 INFO:teuthology.orchestra.run.trial095.stdout:(80/158): lua-5.4.4-4.el9.x86_64.rpm 333 kB/s | 188 kB 00:00 2026-02-26T03:14:09.280 INFO:teuthology.orchestra.run.trial095.stdout:(81/158): python3-devel-3.9.25-3.el9.x86_64.rpm 2.8 MB/s | 244 kB 00:00 2026-02-26T03:14:09.304 INFO:teuthology.orchestra.run.trial095.stdout:(82/158): python3-iniconfig-1.1.1-7.el9.noarch. 179 kB/s | 17 kB 00:00 2026-02-26T03:14:09.366 INFO:teuthology.orchestra.run.trial095.stdout:(83/158): python3-jinja2-2.11.3-8.el9.noarch.rp 2.9 MB/s | 249 kB 00:00 2026-02-26T03:14:09.389 INFO:teuthology.orchestra.run.trial095.stdout:(84/158): python3-jmespath-1.0.1-1.el9.noarch.r 562 kB/s | 48 kB 00:00 2026-02-26T03:14:09.450 INFO:teuthology.orchestra.run.trial095.stdout:(85/158): python3-libstoragemgmt-1.10.1-1.el9.x 2.0 MB/s | 177 kB 00:00 2026-02-26T03:14:09.534 INFO:teuthology.orchestra.run.trial095.stdout:(86/158): python3-markupsafe-1.1.1-12.el9.x86_6 415 kB/s | 35 kB 00:00 2026-02-26T03:14:09.636 INFO:teuthology.orchestra.run.trial095.stdout:(87/158): python3-lxml-4.6.5-3.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2026-02-26T03:14:09.719 INFO:teuthology.orchestra.run.trial095.stdout:(88/158): python3-numpy-1.23.5-2.el9.x86_64.rpm 33 MB/s | 6.1 MB 00:00 2026-02-26T03:14:09.724 INFO:teuthology.orchestra.run.trial095.stdout:(89/158): python3-numpy-f2py-1.23.5-2.el9.x86_6 4.9 MB/s | 442 kB 00:00 2026-02-26T03:14:09.804 INFO:teuthology.orchestra.run.trial095.stdout:(90/158): python3-packaging-20.9-5.el9.noarch.r 908 kB/s | 77 kB 00:00 2026-02-26T03:14:09.818 INFO:teuthology.orchestra.run.trial095.stdout:(91/158): python3-pluggy-0.13.1-7.el9.noarch.rp 441 kB/s | 41 kB 00:00 2026-02-26T03:14:09.890 INFO:teuthology.orchestra.run.trial095.stdout:(92/158): python3-protobuf-3.14.0-17.el9.noarch 3.0 MB/s | 267 kB 00:00 2026-02-26T03:14:09.946 INFO:teuthology.orchestra.run.trial095.stdout:(93/158): python3-py-1.10.0-6.el9.noarch.rpm 3.6 MB/s | 477 kB 00:00 2026-02-26T03:14:09.975 INFO:teuthology.orchestra.run.trial095.stdout:(94/158): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.8 MB/s | 157 kB 00:00 2026-02-26T03:14:10.032 INFO:teuthology.orchestra.run.trial095.stdout:(95/158): python3-pyasn1-modules-0.4.8-7.el9.no 3.2 MB/s | 277 kB 00:00 2026-02-26T03:14:10.079 INFO:teuthology.orchestra.run.trial095.stdout:(96/158): python3-pytest-6.2.2-7.el9.noarch.rpm 4.9 MB/s | 519 kB 00:00 2026-02-26T03:14:10.117 INFO:teuthology.orchestra.run.trial095.stdout:(97/158): python3-requests-oauthlib-1.3.0-12.el 634 kB/s | 54 kB 00:00 2026-02-26T03:14:10.202 INFO:teuthology.orchestra.run.trial095.stdout:(98/158): python3-toml-0.10.2-6.el9.noarch.rpm 492 kB/s | 42 kB 00:00 2026-02-26T03:14:10.287 INFO:teuthology.orchestra.run.trial095.stdout:(99/158): qatlib-25.08.0-2.el9.x86_64.rpm 2.7 MB/s | 240 kB 00:00 2026-02-26T03:14:10.371 INFO:teuthology.orchestra.run.trial095.stdout:(100/158): qatlib-service-25.08.0-2.el9.x86_64. 442 kB/s | 37 kB 00:00 2026-02-26T03:14:10.455 INFO:teuthology.orchestra.run.trial095.stdout:(101/158): qatzip-libs-1.3.1-1.el9.x86_64.rpm 787 kB/s | 66 kB 00:00 2026-02-26T03:14:10.541 INFO:teuthology.orchestra.run.trial095.stdout:(102/158): socat-1.7.4.1-8.el9.x86_64.rpm 3.5 MB/s | 303 kB 00:00 2026-02-26T03:14:10.583 INFO:teuthology.orchestra.run.trial095.stdout:(103/158): python3-scipy-1.9.3-2.el9.x86_64.rpm 38 MB/s | 19 MB 00:00 2026-02-26T03:14:10.627 INFO:teuthology.orchestra.run.trial095.stdout:(104/158): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.1 MB/s | 189 kB 00:00 2026-02-26T03:14:10.667 INFO:teuthology.orchestra.run.trial095.stdout:(105/158): xmlsec1-openssl-1.2.29-13.el9.x86_64 1.1 MB/s | 90 kB 00:00 2026-02-26T03:14:10.711 INFO:teuthology.orchestra.run.trial095.stdout:(106/158): xmlstarlet-1.6.1-20.el9.x86_64.rpm 755 kB/s | 64 kB 00:00 2026-02-26T03:14:11.237 INFO:teuthology.orchestra.run.trial095.stdout:(107/158): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2026-02-26T03:14:11.632 INFO:teuthology.orchestra.run.trial095.stdout:(108/158): abseil-cpp-20211102.0-4.el9.x86_64.r 1.4 MB/s | 551 kB 00:00 2026-02-26T03:14:11.661 INFO:teuthology.orchestra.run.trial095.stdout:(109/158): protobuf-compiler-3.14.0-17.el9.x86_ 908 kB/s | 862 kB 00:00 2026-02-26T03:14:11.703 INFO:teuthology.orchestra.run.trial095.stdout:(110/158): gperftools-libs-2.9.1-3.el9.x86_64.r 4.2 MB/s | 308 kB 00:00 2026-02-26T03:14:11.820 INFO:teuthology.orchestra.run.trial095.stdout:(111/158): grpc-data-1.46.7-10.el9.noarch.rpm 122 kB/s | 19 kB 00:00 2026-02-26T03:14:11.895 INFO:teuthology.orchestra.run.trial095.stdout:(112/158): libarrow-doc-9.0.0-15.el9.noarch.rpm 329 kB/s | 25 kB 00:00 2026-02-26T03:14:11.928 INFO:teuthology.orchestra.run.trial095.stdout:(113/158): libarrow-9.0.0-15.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2026-02-26T03:14:11.981 INFO:teuthology.orchestra.run.trial095.stdout:(114/158): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2026-02-26T03:14:11.994 INFO:teuthology.orchestra.run.trial095.stdout:(115/158): liboath-2.6.12-1.el9.x86_64.rpm 494 kB/s | 49 kB 00:00 2026-02-26T03:14:12.037 INFO:teuthology.orchestra.run.trial095.stdout:(116/158): luarocks-3.9.2-5.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2026-02-26T03:14:12.095 INFO:teuthology.orchestra.run.trial095.stdout:(117/158): python3-asyncssh-2.13.2-5.el9.noarch 9.3 MB/s | 548 kB 00:00 2026-02-26T03:14:12.149 INFO:teuthology.orchestra.run.trial095.stdout:(118/158): python3-autocommand-2.2.2-8.el9.noar 547 kB/s | 29 kB 00:00 2026-02-26T03:14:12.203 INFO:teuthology.orchestra.run.trial095.stdout:(119/158): python3-backports-tarfile-1.2.0-1.el 1.1 MB/s | 60 kB 00:00 2026-02-26T03:14:12.257 INFO:teuthology.orchestra.run.trial095.stdout:(120/158): python3-bcrypt-3.2.2-1.el9.x86_64.rp 809 kB/s | 43 kB 00:00 2026-02-26T03:14:12.291 INFO:teuthology.orchestra.run.trial095.stdout:(121/158): parquet-libs-9.0.0-15.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2026-02-26T03:14:12.310 INFO:teuthology.orchestra.run.trial095.stdout:(122/158): python3-cachetools-4.2.4-1.el9.noarc 605 kB/s | 32 kB 00:00 2026-02-26T03:14:12.344 INFO:teuthology.orchestra.run.trial095.stdout:(123/158): python3-certifi-2023.05.07-4.el9.noa 269 kB/s | 14 kB 00:00 2026-02-26T03:14:12.366 INFO:teuthology.orchestra.run.trial095.stdout:(124/158): python3-cheroot-10.0.1-4.el9.noarch. 3.0 MB/s | 173 kB 00:00 2026-02-26T03:14:12.413 INFO:teuthology.orchestra.run.trial095.stdout:(125/158): python3-cherrypy-18.6.1-2.el9.noarch 5.1 MB/s | 358 kB 00:00 2026-02-26T03:14:12.424 INFO:teuthology.orchestra.run.trial095.stdout:(126/158): python3-google-auth-2.45.0-1.el9.noa 4.3 MB/s | 254 kB 00:00 2026-02-26T03:14:12.481 INFO:teuthology.orchestra.run.trial095.stdout:(127/158): python3-grpcio-tools-1.46.7-10.el9.x 2.5 MB/s | 144 kB 00:00 2026-02-26T03:14:12.537 INFO:teuthology.orchestra.run.trial095.stdout:(128/158): python3-influxdb-5.3.1-1.el9.noarch. 2.4 MB/s | 139 kB 00:00 2026-02-26T03:14:12.564 INFO:teuthology.orchestra.run.trial095.stdout:(129/158): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2026-02-26T03:14:12.591 INFO:teuthology.orchestra.run.trial095.stdout:(130/158): python3-isodate-0.6.1-3.el9.noarch.r 1.0 MB/s | 56 kB 00:00 2026-02-26T03:14:12.616 INFO:teuthology.orchestra.run.trial095.stdout:(131/158): python3-jaraco-8.2.1-3.el9.noarch.rp 204 kB/s | 11 kB 00:00 2026-02-26T03:14:12.645 INFO:teuthology.orchestra.run.trial095.stdout:(132/158): python3-jaraco-classes-3.2.1-5.el9.n 329 kB/s | 18 kB 00:00 2026-02-26T03:14:12.669 INFO:teuthology.orchestra.run.trial095.stdout:(133/158): python3-jaraco-collections-3.0.0-8.e 439 kB/s | 23 kB 00:00 2026-02-26T03:14:12.698 INFO:teuthology.orchestra.run.trial095.stdout:(134/158): python3-jaraco-context-6.0.1-3.el9.n 367 kB/s | 20 kB 00:00 2026-02-26T03:14:12.721 INFO:teuthology.orchestra.run.trial095.stdout:(135/158): python3-jaraco-functools-3.5.0-2.el9 371 kB/s | 19 kB 00:00 2026-02-26T03:14:12.752 INFO:teuthology.orchestra.run.trial095.stdout:(136/158): python3-jaraco-text-4.0.0-2.el9.noar 492 kB/s | 26 kB 00:00 2026-02-26T03:14:12.790 INFO:teuthology.orchestra.run.trial095.stdout:(137/158): python3-kubernetes-26.1.0-3.el9.noar 15 MB/s | 1.0 MB 00:00 2026-02-26T03:14:12.807 INFO:teuthology.orchestra.run.trial095.stdout:(138/158): python3-more-itertools-8.12.0-2.el9. 1.4 MB/s | 79 kB 00:00 2026-02-26T03:14:12.844 INFO:teuthology.orchestra.run.trial095.stdout:(139/158): python3-msgpack-1.0.3-2.el9.x86_64.r 1.6 MB/s | 86 kB 00:00 2026-02-26T03:14:12.861 INFO:teuthology.orchestra.run.trial095.stdout:(140/158): python3-natsort-7.1.1-5.el9.noarch.r 1.0 MB/s | 58 kB 00:00 2026-02-26T03:14:12.897 INFO:teuthology.orchestra.run.trial095.stdout:(141/158): python3-portend-3.1.0-2.el9.noarch.r 313 kB/s | 16 kB 00:00 2026-02-26T03:14:12.916 INFO:teuthology.orchestra.run.trial095.stdout:(142/158): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.6 MB/s | 90 kB 00:00 2026-02-26T03:14:12.949 INFO:teuthology.orchestra.run.trial095.stdout:(143/158): python3-repoze-lru-0.7-16.el9.noarch 582 kB/s | 31 kB 00:00 2026-02-26T03:14:12.972 INFO:teuthology.orchestra.run.trial095.stdout:(144/158): python3-routes-2.5.1-5.el9.noarch.rp 3.3 MB/s | 188 kB 00:00 2026-02-26T03:14:13.002 INFO:teuthology.orchestra.run.trial095.stdout:(145/158): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2026-02-26T03:14:13.028 INFO:teuthology.orchestra.run.trial095.stdout:(146/158): python3-saml-1.16.0-1.el9.noarch.rpm 2.2 MB/s | 125 kB 00:00 2026-02-26T03:14:13.055 INFO:teuthology.orchestra.run.trial095.stdout:(147/158): python3-tempora-5.0.0-2.el9.noarch.r 676 kB/s | 36 kB 00:00 2026-02-26T03:14:13.083 INFO:teuthology.orchestra.run.trial095.stdout:(148/158): python3-typing-extensions-4.15.0-1.e 1.5 MB/s | 86 kB 00:00 2026-02-26T03:14:13.109 INFO:teuthology.orchestra.run.trial095.stdout:(149/158): python3-websocket-client-1.2.3-2.el9 1.6 MB/s | 90 kB 00:00 2026-02-26T03:14:13.137 INFO:teuthology.orchestra.run.trial095.stdout:(150/158): python3-xmlsec-1.3.13-1.el9.x86_64.r 886 kB/s | 48 kB 00:00 2026-02-26T03:14:13.161 INFO:teuthology.orchestra.run.trial095.stdout:(151/158): python3-xmltodict-0.12.0-15.el9.noar 420 kB/s | 22 kB 00:00 2026-02-26T03:14:13.191 INFO:teuthology.orchestra.run.trial095.stdout:(152/158): python3-zc-lockfile-2.0-10.el9.noarc 373 kB/s | 20 kB 00:00 2026-02-26T03:14:13.215 INFO:teuthology.orchestra.run.trial095.stdout:(153/158): re2-20211101-20.el9.x86_64.rpm 3.5 MB/s | 191 kB 00:00 2026-02-26T03:14:13.257 INFO:teuthology.orchestra.run.trial095.stdout:(154/158): thrift-0.15.0-4.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2026-02-26T03:14:13.269 INFO:teuthology.orchestra.run.trial095.stdout:(155/158): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.2 MB/s | 122 kB 00:00 2026-02-26T03:14:15.113 INFO:teuthology.orchestra.run.trial095.stdout:(156/158): librbd1-20.3.0-5312.gabffb617.el9.x8 1.5 MB/s | 2.7 MB 00:01 2026-02-26T03:14:17.552 INFO:teuthology.orchestra.run.trial095.stdout:(157/158): librados2-20.3.0-5312.gabffb617.el9. 783 kB/s | 3.3 MB 00:04 2026-02-26T03:23:47.654 INFO:teuthology.orchestra.run.trial095.stdout:(158/158): ceph-osd-crimson-debuginfo-20.3.0-53 638 kB/s | 429 MB 11:28 2026-02-26T03:23:47.662 INFO:teuthology.orchestra.run.trial095.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:23:47.662 INFO:teuthology.orchestra.run.trial095.stdout:Total 1.2 MB/s | 863 MB 11:42 2026-02-26T03:23:48.193 INFO:teuthology.orchestra.run.trial095.stdout:Running transaction check 2026-02-26T03:23:48.247 INFO:teuthology.orchestra.run.trial095.stdout:Transaction check succeeded. 2026-02-26T03:23:48.248 INFO:teuthology.orchestra.run.trial095.stdout:Running transaction test 2026-02-26T03:23:50.998 INFO:teuthology.orchestra.run.trial095.stdout:Transaction test succeeded. 2026-02-26T03:23:50.999 INFO:teuthology.orchestra.run.trial095.stdout:Running transaction 2026-02-26T03:23:53.825 INFO:teuthology.orchestra.run.trial095.stdout: Preparing : 1/1 2026-02-26T03:23:53.828 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/160 2026-02-26T03:23:53.836 INFO:teuthology.orchestra.run.trial095.stdout: Installing : liboath-2.6.12-1.el9.x86_64 2/160 2026-02-26T03:23:53.847 INFO:teuthology.orchestra.run.trial095.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/160 2026-02-26T03:23:53.990 INFO:teuthology.orchestra.run.trial095.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/160 2026-02-26T03:23:53.991 INFO:teuthology.orchestra.run.trial095.stdout: Upgrading : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 5/160 2026-02-26T03:23:54.038 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 5/160 2026-02-26T03:23:54.039 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 6/160 2026-02-26T03:23:54.050 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 6/160 2026-02-26T03:23:54.052 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86 7/160 2026-02-26T03:23:54.053 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86 8/160 2026-02-26T03:23:54.080 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86 8/160 2026-02-26T03:23:54.086 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 9/160 2026-02-26T03:23:54.094 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 10/160 2026-02-26T03:23:54.096 INFO:teuthology.orchestra.run.trial095.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 11/160 2026-02-26T03:23:54.099 INFO:teuthology.orchestra.run.trial095.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 12/160 2026-02-26T03:23:54.101 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 13/160 2026-02-26T03:23:54.105 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 14/160 2026-02-26T03:23:54.221 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 15/160 2026-02-26T03:23:54.222 INFO:teuthology.orchestra.run.trial095.stdout: Upgrading : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 16/160 2026-02-26T03:23:54.269 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 16/160 2026-02-26T03:23:54.275 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 17/160 2026-02-26T03:23:54.285 INFO:teuthology.orchestra.run.trial095.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 18/160 2026-02-26T03:23:54.285 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 19/160 2026-02-26T03:23:54.307 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 19/160 2026-02-26T03:23:54.308 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86 20/160 2026-02-26T03:23:54.319 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86 20/160 2026-02-26T03:23:54.323 INFO:teuthology.orchestra.run.trial095.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 21/160 2026-02-26T03:23:54.354 INFO:teuthology.orchestra.run.trial095.stdout: Installing : re2-1:20211101-20.el9.x86_64 22/160 2026-02-26T03:23:54.360 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 23/160 2026-02-26T03:23:54.363 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 24/160 2026-02-26T03:23:54.385 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 25/160 2026-02-26T03:23:54.390 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 26/160 2026-02-26T03:23:54.394 INFO:teuthology.orchestra.run.trial095.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/160 2026-02-26T03:23:54.398 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 28/160 2026-02-26T03:23:54.399 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 29/160 2026-02-26T03:23:54.411 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 29/160 2026-02-26T03:23:54.417 INFO:teuthology.orchestra.run.trial095.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 30/160 2026-02-26T03:23:54.442 INFO:teuthology.orchestra.run.trial095.stdout: Installing : unzip-6.0-59.el9.x86_64 31/160 2026-02-26T03:23:54.455 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 32/160 2026-02-26T03:23:54.461 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 33/160 2026-02-26T03:23:54.465 INFO:teuthology.orchestra.run.trial095.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 34/160 2026-02-26T03:23:54.471 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 35/160 2026-02-26T03:23:54.473 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 36/160 2026-02-26T03:23:54.474 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 37/160 2026-02-26T03:23:54.484 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 38/160 2026-02-26T03:23:54.519 INFO:teuthology.orchestra.run.trial095.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 39/160 2026-02-26T03:23:54.523 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-ceph-common-2:20.3.0-5312.gabffb617.el9. 40/160 2026-02-26T03:23:54.534 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el 41/160 2026-02-26T03:23:54.545 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_6 42/160 2026-02-26T03:23:54.551 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 43/160 2026-02-26T03:23:54.576 INFO:teuthology.orchestra.run.trial095.stdout: Installing : zip-3.0-35.el9.x86_64 44/160 2026-02-26T03:23:54.579 INFO:teuthology.orchestra.run.trial095.stdout: Installing : luarocks-3.9.2-5.el9.noarch 45/160 2026-02-26T03:23:54.629 INFO:teuthology.orchestra.run.trial095.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 46/160 2026-02-26T03:23:54.642 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 47/160 2026-02-26T03:23:54.650 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/160 2026-02-26T03:23:54.653 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/160 2026-02-26T03:23:54.655 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/160 2026-02-26T03:23:54.661 INFO:teuthology.orchestra.run.trial095.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 51/160 2026-02-26T03:23:54.677 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 52/160 2026-02-26T03:23:54.683 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 53/160 2026-02-26T03:23:54.686 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/160 2026-02-26T03:23:54.693 INFO:teuthology.orchestra.run.trial095.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 55/160 2026-02-26T03:23:54.701 INFO:teuthology.orchestra.run.trial095.stdout: Installing : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_6 56/160 2026-02-26T03:23:54.713 INFO:teuthology.orchestra.run.trial095.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 57/160 2026-02-26T03:23:54.717 INFO:teuthology.orchestra.run.trial095.stdout: Installing : thrift-0.15.0-4.el9.x86_64 58/160 2026-02-26T03:23:54.721 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 59/160 2026-02-26T03:23:54.735 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 60/160 2026-02-26T03:23:54.740 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 61/160 2026-02-26T03:23:54.745 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 62/160 2026-02-26T03:23:54.756 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 63/160 2026-02-26T03:23:54.765 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 64/160 2026-02-26T03:23:54.771 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 65/160 2026-02-26T03:23:54.791 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 66/160 2026-02-26T03:23:54.803 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 67/160 2026-02-26T03:23:54.818 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 68/160 2026-02-26T03:23:54.822 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 69/160 2026-02-26T03:23:54.830 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 70/160 2026-02-26T03:23:54.870 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 71/160 2026-02-26T03:23:55.195 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 72/160 2026-02-26T03:23:55.205 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 73/160 2026-02-26T03:23:55.209 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 74/160 2026-02-26T03:23:55.215 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 75/160 2026-02-26T03:23:55.220 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 76/160 2026-02-26T03:23:55.224 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 77/160 2026-02-26T03:23:55.228 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 78/160 2026-02-26T03:23:55.234 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 79/160 2026-02-26T03:23:55.237 INFO:teuthology.orchestra.run.trial095.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 80/160 2026-02-26T03:23:55.238 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 81/160 2026-02-26T03:23:55.268 INFO:teuthology.orchestra.run.trial095.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 82/160 2026-02-26T03:23:55.313 INFO:teuthology.orchestra.run.trial095.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 83/160 2026-02-26T03:23:55.322 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 84/160 2026-02-26T03:23:55.369 INFO:teuthology.orchestra.run.trial095.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 85/160 2026-02-26T03:23:55.398 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-py-1.10.0-6.el9.noarch 86/160 2026-02-26T03:23:55.408 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 87/160 2026-02-26T03:23:55.414 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 88/160 2026-02-26T03:23:55.420 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-pluggy-0.13.1-7.el9.noarch 89/160 2026-02-26T03:23:55.427 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 90/160 2026-02-26T03:23:55.431 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 91/160 2026-02-26T03:23:55.465 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-iniconfig-1.1.1-7.el9.noarch 92/160 2026-02-26T03:23:55.690 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 93/160 2026-02-26T03:23:55.713 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 94/160 2026-02-26T03:23:55.716 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 95/160 2026-02-26T03:23:55.775 INFO:teuthology.orchestra.run.trial095.stdout: Installing : openblas-0.3.29-1.el9.x86_64 96/160 2026-02-26T03:23:55.776 INFO:teuthology.orchestra.run.trial095.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 97/160 2026-02-26T03:23:55.797 INFO:teuthology.orchestra.run.trial095.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 98/160 2026-02-26T03:23:56.123 INFO:teuthology.orchestra.run.trial095.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 99/160 2026-02-26T03:23:56.196 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 100/160 2026-02-26T03:23:56.869 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 101/160 2026-02-26T03:23:56.884 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 102/160 2026-02-26T03:23:56.890 INFO:teuthology.orchestra.run.trial095.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 103/160 2026-02-26T03:23:57.140 INFO:teuthology.orchestra.run.trial095.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 104/160 2026-02-26T03:23:57.141 INFO:teuthology.orchestra.run.trial095.stdout: Installing : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 105/160 2026-02-26T03:23:57.159 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 105/160 2026-02-26T03:23:57.160 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 106/160 2026-02-26T03:23:58.100 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 107/160 2026-02-26T03:23:58.102 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 107/160 2026-02-26T03:23:58.120 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 107/160 2026-02-26T03:23:58.131 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 108/160 2026-02-26T03:23:58.138 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-packaging-20.9-5.el9.noarch 109/160 2026-02-26T03:23:58.152 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-ply-3.11-14.el9.noarch 110/160 2026-02-26T03:23:58.168 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 111/160 2026-02-26T03:23:58.244 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 112/160 2026-02-26T03:23:58.252 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 113/160 2026-02-26T03:23:58.276 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 114/160 2026-02-26T03:23:58.306 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 115/160 2026-02-26T03:23:58.357 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 116/160 2026-02-26T03:23:58.363 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 117/160 2026-02-26T03:23:58.366 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 118/160 2026-02-26T03:23:58.372 INFO:teuthology.orchestra.run.trial095.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 119/160 2026-02-26T03:23:58.374 INFO:teuthology.orchestra.run.trial095.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 120/160 2026-02-26T03:23:58.375 INFO:teuthology.orchestra.run.trial095.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 121/160 2026-02-26T03:23:58.391 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 121/160 2026-02-26T03:23:58.509 INFO:teuthology.orchestra.run.trial095.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 122/160 2026-02-26T03:23:58.511 INFO:teuthology.orchestra.run.trial095.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 123/160 2026-02-26T03:23:58.796 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 123/160 2026-02-26T03:23:58.796 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T03:23:58.797 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:23:59.055 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 124/160 2026-02-26T03:23:59.095 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 124/160 2026-02-26T03:23:59.095 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:23:59.096 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:23:59.096 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:23:59.100 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 125/160 2026-02-26T03:24:03.771 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 125/160 2026-02-26T03:24:03.771 INFO:teuthology.orchestra.run.trial095.stdout:skipping the directory /sys 2026-02-26T03:24:03.771 INFO:teuthology.orchestra.run.trial095.stdout:skipping the directory /proc 2026-02-26T03:24:03.771 INFO:teuthology.orchestra.run.trial095.stdout:skipping the directory /mnt 2026-02-26T03:24:03.771 INFO:teuthology.orchestra.run.trial095.stdout:skipping the directory /var/tmp 2026-02-26T03:24:03.772 INFO:teuthology.orchestra.run.trial095.stdout:skipping the directory /home 2026-02-26T03:24:03.772 INFO:teuthology.orchestra.run.trial095.stdout:skipping the directory /root 2026-02-26T03:24:03.772 INFO:teuthology.orchestra.run.trial095.stdout:skipping the directory /tmp 2026-02-26T03:24:03.772 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:03.859 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 126/160 2026-02-26T03:24:03.880 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 126/160 2026-02-26T03:24:03.880 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:03.881 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:24:03.881 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:24:03.881 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:24:03.881 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:04.086 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 127/160 2026-02-26T03:24:04.106 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 127/160 2026-02-26T03:24:04.106 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:04.106 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:24:04.107 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:24:04.107 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:24:04.107 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:04.113 INFO:teuthology.orchestra.run.trial095.stdout: Installing : mailcap-2.1.49-5.el9.noarch 128/160 2026-02-26T03:24:04.114 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 129/160 2026-02-26T03:24:04.131 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 130/160 2026-02-26T03:24:04.131 INFO:teuthology.orchestra.run.trial095.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:24:04.131 INFO:teuthology.orchestra.run.trial095.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:24:04.131 INFO:teuthology.orchestra.run.trial095.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:24:04.131 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:04.138 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 130/160 2026-02-26T03:24:04.161 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 130/160 2026-02-26T03:24:04.161 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:24:04.161 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:04.193 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 131/160 2026-02-26T03:24:04.779 INFO:teuthology.orchestra.run.trial095.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 132/160 2026-02-26T03:24:10.102 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86 133/160 2026-02-26T03:24:10.164 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86 134/160 2026-02-26T03:24:10.166 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 135/160 2026-02-26T03:24:10.179 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 135/160 2026-02-26T03:24:10.179 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:10.179 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:24:10.179 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:10.189 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 136/160 2026-02-26T03:24:10.211 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 136/160 2026-02-26T03:24:10.211 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:10.212 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:24:10.212 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:24:10.212 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:24:10.212 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:10.270 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 137/160 2026-02-26T03:24:10.272 INFO:teuthology.orchestra.run.trial095.stdout: Installing : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 137/160 2026-02-26T03:24:10.281 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.e 138/160 2026-02-26T03:24:10.304 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617. 139/160 2026-02-26T03:24:10.306 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noa 140/160 2026-02-26T03:24:13.175 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noa 140/160 2026-02-26T03:24:13.182 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.n 141/160 2026-02-26T03:24:13.634 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.n 141/160 2026-02-26T03:24:13.635 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabf 142/160 2026-02-26T03:24:13.646 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabf 142/160 2026-02-26T03:24:13.647 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.n 143/160 2026-02-26T03:24:13.708 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.n 143/160 2026-02-26T03:24:13.748 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el 144/160 2026-02-26T03:24:13.749 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 145/160 2026-02-26T03:24:13.768 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 145/160 2026-02-26T03:24:13.768 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:13.768 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:24:13.769 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:24:13.769 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:24:13.769 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:13.781 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 146/160 2026-02-26T03:24:16.034 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 146/160 2026-02-26T03:24:38.080 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86 147/160 2026-02-26T03:24:38.081 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb6 148/160 2026-02-26T03:24:38.564 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 149/160 2026-02-26T03:24:38.565 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 150/160 2026-02-26T03:24:38.587 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 150/160 2026-02-26T03:24:38.587 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:38.587 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:24:38.587 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:24:38.588 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:24:38.588 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:38.597 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.gabffb 151/160 2026-02-26T03:24:38.616 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.gabffb 151/160 2026-02-26T03:24:38.616 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:38.616 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:24:38.617 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:38.726 INFO:teuthology.orchestra.run.trial095.stdout: Installing : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 152/160 2026-02-26T03:24:38.746 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 152/160 2026-02-26T03:24:38.746 INFO:teuthology.orchestra.run.trial095.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:38.747 INFO:teuthology.orchestra.run.trial095.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:24:38.747 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:24:38.747 INFO:teuthology.orchestra.run.trial095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:24:38.747 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:45.680 INFO:teuthology.orchestra.run.trial095.stdout: Installing : python3-pytest-6.2.2-7.el9.noarch 153/160 2026-02-26T03:24:45.688 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 154/160 2026-02-26T03:24:45.693 INFO:teuthology.orchestra.run.trial095.stdout: Installing : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_ 155/160 2026-02-26T03:24:45.702 INFO:teuthology.orchestra.run.trial095.stdout: Installing : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 156/160 2026-02-26T03:24:45.742 INFO:teuthology.orchestra.run.trial095.stdout: Installing : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 157/160 2026-02-26T03:24:45.745 INFO:teuthology.orchestra.run.trial095.stdout: Installing : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 158/160 2026-02-26T03:24:45.745 INFO:teuthology.orchestra.run.trial095.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/160 2026-02-26T03:24:45.758 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/160 2026-02-26T03:24:45.758 INFO:teuthology.orchestra.run.trial095.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:45.770 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:45.776 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86 160/160 2026-02-26T03:24:45.801 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86 160/160 2026-02-26T03:24:48.867 INFO:teuthology.orchestra.run.trial095.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 1/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 3/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86 4/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 5/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.gabffb 6/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 7/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 8/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 9/160 2026-02-26T03:24:48.868 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 10/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86 11/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86 12/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb6 13/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 14/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 15/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 16/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86 17/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_ 18/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86 19/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 20/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 21/160 2026-02-26T03:24:48.869 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_6 22/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86 23/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 24/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el 25/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.gabffb617.el9. 26/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_6 27/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 28/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 29/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 30/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 31/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 32/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 33/160 2026-02-26T03:24:48.870 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617. 34/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noa 35/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.n 36/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabf 37/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.n 38/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el 39/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 40/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.e 41/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 42/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 43/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 44/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 45/160 2026-02-26T03:24:48.871 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 46/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 47/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 48/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 49/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 50/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 51/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 52/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 53/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 54/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 55/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 56/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-ply-3.11-14.el9.noarch 57/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 58/160 2026-02-26T03:24:48.872 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 59/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 60/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 61/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : unzip-6.0-59.el9.x86_64 62/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : zip-3.0-35.el9.x86_64 63/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 64/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 65/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 66/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 67/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 68/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 69/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/160 2026-02-26T03:24:48.873 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 73/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 74/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 75/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 76/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : lua-5.4.4-4.el9.x86_64 77/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 78/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 79/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 80/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 81/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 82/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-iniconfig-1.1.1-7.el9.noarch 83/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 84/160 2026-02-26T03:24:48.874 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 85/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 86/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 87/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 88/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 89/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 90/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 91/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-pluggy-0.13.1-7.el9.noarch 92/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 93/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-py-1.10.0-6.el9.noarch 94/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 95/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 96/160 2026-02-26T03:24:48.875 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-pytest-6.2.2-7.el9.noarch 97/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 98/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 99/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 100/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 101/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 102/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 103/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 104/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 105/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 106/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 107/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 108/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 109/160 2026-02-26T03:24:48.876 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 110/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 111/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 112/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 113/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 114/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 115/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 116/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 117/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 118/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 119/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 120/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 121/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 122/160 2026-02-26T03:24:48.877 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 123/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 124/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 125/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 126/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 127/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 128/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 129/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 130/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 131/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 132/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 133/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 134/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 135/160 2026-02-26T03:24:48.878 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 136/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 137/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 138/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 139/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 140/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 141/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 142/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 143/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 144/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 145/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 146/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 147/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 148/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 149/160 2026-02-26T03:24:48.879 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 150/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 151/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 152/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 153/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : re2-1:20211101-20.el9.x86_64 154/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 155/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 156/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 157/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 158/160 2026-02-26T03:24:48.880 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 159/160 2026-02-26T03:24:48.941 INFO:teuthology.orchestra.run.trial095.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout:Upgraded: 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout:Installed: 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.942 INFO:teuthology.orchestra.run.trial095.stdout: ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.943 INFO:teuthology.orchestra.run.trial095.stdout: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:24:48.944 INFO:teuthology.orchestra.run.trial095.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.945 INFO:teuthology.orchestra.run.trial095.stdout: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.946 INFO:teuthology.orchestra.run.trial095.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:24:48.947 INFO:teuthology.orchestra.run.trial095.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.948 INFO:teuthology.orchestra.run.trial095.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-iniconfig-1.1.1-7.el9.noarch 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:24:48.949 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:24:48.950 INFO:teuthology.orchestra.run.trial095.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-pluggy-0.13.1-7.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-py-1.10.0-6.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:24:48.951 INFO:teuthology.orchestra.run.trial095.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-pytest-6.2.2-7.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:24:48.952 INFO:teuthology.orchestra.run.trial095.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.953 INFO:teuthology.orchestra.run.trial095.stdout: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:24:48.954 INFO:teuthology.orchestra.run.trial095.stdout:Complete! 2026-02-26T03:24:49.095 DEBUG:teuthology.parallel:result is None 2026-02-26T03:24:49.096 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:24:49.263 DEBUG:teuthology.orchestra.run.trial095:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:24:49.286 INFO:teuthology.orchestra.run.trial095.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:24:49.286 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:24:49.286 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:24:49.288 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:24:49.289 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:24:49.289 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:24:49.353 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:24:49.354 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:24:49.354 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:24:49.417 DEBUG:teuthology.orchestra.run.trial095:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:24:49.480 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:24:49.480 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:24:49.480 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:24:49.543 DEBUG:teuthology.orchestra.run.trial095:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:24:49.607 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:24:49.607 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:24:49.607 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:24:49.671 DEBUG:teuthology.orchestra.run.trial095:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:24:49.735 INFO:teuthology.run_tasks:Running task pexec... 2026-02-26T03:24:49.750 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-26T03:24:49.751 DEBUG:teuthology.orchestra.run.trial095:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-26T03:24:49.777 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:24:49.778 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-26T03:24:49.827 DEBUG:teuthology.parallel:result is None 2026-02-26T03:24:49.827 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:24:49.842 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:24:49.842 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:24:49.842 DEBUG:teuthology.task.install:config {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:24:49.842 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:24:49.851 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:24:49.851 INFO:teuthology.task.install:extra packages: [] 2026-02-26T03:24:49.852 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:24:49.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:24:50.011 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:24:50.012 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:24:50.392 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:24:50.392 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:24:50.392 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:24:50.417 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-pytest on remote rpm x86_64 2026-02-26T03:24:50.417 DEBUG:teuthology.orchestra.run.trial095:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:24:50.482 DEBUG:teuthology.orchestra.run.trial095:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:24:50.546 DEBUG:teuthology.orchestra.run.trial095:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:24:50.601 INFO:teuthology.orchestra.run.trial095.stdout:check_obsoletes = 1 2026-02-26T03:24:50.610 DEBUG:teuthology.orchestra.run.trial095:> sudo yum clean all 2026-02-26T03:24:50.771 INFO:teuthology.orchestra.run.trial095.stdout:56 files removed 2026-02-26T03:24:50.786 DEBUG:teuthology.orchestra.run.trial095:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest 2026-02-26T03:24:51.669 INFO:teuthology.orchestra.run.trial095.stdout:ceph packages for x86_64 129 kB/s | 94 kB 00:00 2026-02-26T03:24:52.345 INFO:teuthology.orchestra.run.trial095.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-26T03:24:52.949 INFO:teuthology.orchestra.run.trial095.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:24:53.819 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.9 MB 00:00 2026-02-26T03:24:56.973 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-26T03:25:00.429 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 8.0 MB 00:01 2026-02-26T03:25:02.064 INFO:teuthology.orchestra.run.trial095.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 20 kB 00:00 2026-02-26T03:25:02.379 INFO:teuthology.orchestra.run.trial095.stdout:Extra Packages for Enterprise Linux 81 MB/s | 20 MB 00:00 2026-02-26T03:25:05.641 INFO:teuthology.orchestra.run.trial095.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-26T03:25:06.835 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.836 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.836 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.837 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.837 INFO:teuthology.orchestra.run.trial095.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:25:06.837 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.837 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.838 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:25:06.838 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:25:06.838 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:25:06.839 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:25:06.839 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.840 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.840 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.841 INFO:teuthology.orchestra.run.trial095.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:25:06.841 INFO:teuthology.orchestra.run.trial095.stdout:Package librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.842 INFO:teuthology.orchestra.run.trial095.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.842 INFO:teuthology.orchestra.run.trial095.stdout:Package libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.842 INFO:teuthology.orchestra.run.trial095.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.843 INFO:teuthology.orchestra.run.trial095.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.843 INFO:teuthology.orchestra.run.trial095.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.844 INFO:teuthology.orchestra.run.trial095.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.844 INFO:teuthology.orchestra.run.trial095.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.845 INFO:teuthology.orchestra.run.trial095.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.845 INFO:teuthology.orchestra.run.trial095.stdout:Package rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.845 INFO:teuthology.orchestra.run.trial095.stdout:Package rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.846 INFO:teuthology.orchestra.run.trial095.stdout:Package rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:25:06.846 INFO:teuthology.orchestra.run.trial095.stdout:Package python3-pytest-6.2.2-7.el9.noarch is already installed. 2026-02-26T03:25:06.865 INFO:teuthology.orchestra.run.trial095.stdout:Dependencies resolved. 2026-02-26T03:25:06.865 INFO:teuthology.orchestra.run.trial095.stdout:Nothing to do. 2026-02-26T03:25:06.865 INFO:teuthology.orchestra.run.trial095.stdout:Complete! 2026-02-26T03:25:06.893 DEBUG:teuthology.parallel:result is None 2026-02-26T03:25:06.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:25:07.048 DEBUG:teuthology.orchestra.run.trial095:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:25:07.069 INFO:teuthology.orchestra.run.trial095.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:25:07.069 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:25:07.069 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:25:07.071 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-26T03:25:07.071 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:25:07.169 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:25:07.170 DEBUG:teuthology.orchestra.run.trial095:> sudo chmod 777 /var/log/ceph 2026-02-26T03:25:07.195 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:25:07.195 DEBUG:teuthology.orchestra.run.trial095:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:25:07.259 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:25:07.259 DEBUG:teuthology.orchestra.run.trial095:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:25:07.332 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:25:07.332 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:25:07.332 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71174', 'branch': 'main', 'description': 'crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} tasks/rbd_python_api_tests}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '71174', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5117, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996', 'targets': {'trial095.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB7arm2FG3dyz5mi5X7eO7jbnddcCgGahAOiLWP3mLYxQspKKPF2KHv4ooJ93kooQKJVq8NpM3ctPKSF4gExYS0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_librbd_python.sh']}, 'env': {'RBD_FEATURES': '61'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_02:22:56', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:25:07.333 DEBUG:teuthology.orchestra.run.trial095:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:25:07.379 DEBUG:teuthology.orchestra.run.trial095:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:25:07.444 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:07.444 DEBUG:teuthology.orchestra.run.trial095:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:25:07.499 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:25:07.500 DEBUG:teuthology.orchestra.run.trial095:> stat /dev/vg_nvme/lv_1 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout:Device: 6h/6d Inode: 1303 Links: 1 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout:Access: 2026-02-26 03:24:46.011348570 +0000 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout:Modify: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout:Change: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:07.557 INFO:teuthology.orchestra.run.trial095.stdout: Birth: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:07.558 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:25:07.619 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records in 2026-02-26T03:25:07.619 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records out 2026-02-26T03:25:07.619 INFO:teuthology.orchestra.run.trial095.stderr:512 bytes copied, 0.00015114 s, 3.4 MB/s 2026-02-26T03:25:07.621 DEBUG:teuthology.orchestra.run.trial095:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:25:07.678 DEBUG:teuthology.orchestra.run.trial095:> stat /dev/vg_nvme/lv_2 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout:Device: 6h/6d Inode: 1307 Links: 1 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout:Access: 2026-02-26 03:24:46.011348570 +0000 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout:Modify: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:07.734 INFO:teuthology.orchestra.run.trial095.stdout:Change: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:07.735 INFO:teuthology.orchestra.run.trial095.stdout: Birth: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:07.735 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:25:07.797 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records in 2026-02-26T03:25:07.797 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records out 2026-02-26T03:25:07.797 INFO:teuthology.orchestra.run.trial095.stderr:512 bytes copied, 0.000196004 s, 2.6 MB/s 2026-02-26T03:25:07.798 DEBUG:teuthology.orchestra.run.trial095:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:25:07.855 DEBUG:teuthology.orchestra.run.trial095:> stat /dev/vg_nvme/lv_3 2026-02-26T03:25:07.911 INFO:teuthology.orchestra.run.trial095.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:25:07.911 INFO:teuthology.orchestra.run.trial095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:25:07.911 INFO:teuthology.orchestra.run.trial095.stdout:Device: 6h/6d Inode: 1293 Links: 1 2026-02-26T03:25:07.911 INFO:teuthology.orchestra.run.trial095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:25:07.911 INFO:teuthology.orchestra.run.trial095.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:25:07.912 INFO:teuthology.orchestra.run.trial095.stdout:Access: 2026-02-26 03:24:46.011348570 +0000 2026-02-26T03:25:07.912 INFO:teuthology.orchestra.run.trial095.stdout:Modify: 2026-02-26 03:23:58.820273441 +0000 2026-02-26T03:25:07.912 INFO:teuthology.orchestra.run.trial095.stdout:Change: 2026-02-26 03:23:58.820273441 +0000 2026-02-26T03:25:07.912 INFO:teuthology.orchestra.run.trial095.stdout: Birth: 2026-02-26 03:23:58.820273441 +0000 2026-02-26T03:25:07.912 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:25:07.975 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records in 2026-02-26T03:25:07.975 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records out 2026-02-26T03:25:07.976 INFO:teuthology.orchestra.run.trial095.stderr:512 bytes copied, 0.000143276 s, 3.6 MB/s 2026-02-26T03:25:07.977 DEBUG:teuthology.orchestra.run.trial095:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:25:08.034 DEBUG:teuthology.orchestra.run.trial095:> stat /dev/vg_nvme/lv_4 2026-02-26T03:25:08.090 INFO:teuthology.orchestra.run.trial095.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:25:08.090 INFO:teuthology.orchestra.run.trial095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:25:08.090 INFO:teuthology.orchestra.run.trial095.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-26T03:25:08.090 INFO:teuthology.orchestra.run.trial095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:25:08.090 INFO:teuthology.orchestra.run.trial095.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:25:08.091 INFO:teuthology.orchestra.run.trial095.stdout:Access: 2026-02-26 03:24:46.012348550 +0000 2026-02-26T03:25:08.091 INFO:teuthology.orchestra.run.trial095.stdout:Modify: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:08.091 INFO:teuthology.orchestra.run.trial095.stdout:Change: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:08.091 INFO:teuthology.orchestra.run.trial095.stdout: Birth: 2026-02-26 03:23:58.821273421 +0000 2026-02-26T03:25:08.091 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:25:08.152 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records in 2026-02-26T03:25:08.153 INFO:teuthology.orchestra.run.trial095.stderr:1+0 records out 2026-02-26T03:25:08.153 INFO:teuthology.orchestra.run.trial095.stderr:512 bytes copied, 0.000225449 s, 2.3 MB/s 2026-02-26T03:25:08.154 DEBUG:teuthology.orchestra.run.trial095:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:25:08.211 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:25:08.211 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial095.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:25:08.211 INFO:tasks.ceph:Generating config... 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-26T03:25:08.213 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-26T03:25:08.214 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-26T03:25:08.215 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-26T03:25:08.216 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:25:08.216 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-26T03:25:08.216 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-26T03:25:08.216 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-26T03:25:08.216 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-26T03:25:08.216 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-26T03:25:08.216 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-26T03:25:08.216 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:25:08.216 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:25:08.216 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:25:08.217 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:25:08.291 INFO:teuthology.orchestra.run.trial095.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:25:08.294 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:25:08.377 DEBUG:teuthology.orchestra.run.trial095:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:25:08.442 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.95')] 2026-02-26T03:25:08.442 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.95', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:25:08.444 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:08.444 DEBUG:teuthology.orchestra.run.trial095:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:25:08.498 DEBUG:teuthology.orchestra.run.trial095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.95 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:08.578 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:08.575+0000 7f8ee2633f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:08.578 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:08.575+0000 7f8ee2633f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:08.578 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:08.575+0000 7f8ee2633f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:08.579 INFO:teuthology.orchestra.run.trial095.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:08.579 INFO:teuthology.orchestra.run.trial095.stdout:monmaptool: generated fsid efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e 2026-02-26T03:25:08.579 INFO:teuthology.orchestra.run.trial095.stdout:setting min_mon_release = tentacle 2026-02-26T03:25:08.579 INFO:teuthology.orchestra.run.trial095.stdout:epoch 0 2026-02-26T03:25:08.580 INFO:teuthology.orchestra.run.trial095.stdout:fsid efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e 2026-02-26T03:25:08.580 INFO:teuthology.orchestra.run.trial095.stdout:last_changed 2026-02-26T03:25:08.577184+0000 2026-02-26T03:25:08.580 INFO:teuthology.orchestra.run.trial095.stdout:created 2026-02-26T03:25:08.577184+0000 2026-02-26T03:25:08.580 INFO:teuthology.orchestra.run.trial095.stdout:min_mon_release 20 (tentacle) 2026-02-26T03:25:08.580 INFO:teuthology.orchestra.run.trial095.stdout:election_strategy: 1 2026-02-26T03:25:08.580 INFO:teuthology.orchestra.run.trial095.stdout:0: [v2:10.20.193.95:3300/0,v1:10.20.193.95:6789/0] mon.a 2026-02-26T03:25:08.580 INFO:teuthology.orchestra.run.trial095.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:25:08.581 DEBUG:teuthology.orchestra.run.trial095:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:25:08.635 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e... 2026-02-26T03:25:08.637 DEBUG:teuthology.orchestra.run.trial095:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout:[global] 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: chdir = "" 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: auth supported = cephx 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: filestore xattr use omap = true 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: mon clock drift allowed = 1.000 2026-02-26T03:25:08.715 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: osd crush chooseleaf type = 0 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: auth debug = true 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: ms die on old message = true 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: ms die on bug = true 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: mon pg warn max object skew = 0 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:25:08.716 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: osd pool default size = 2 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon osd allow primary affinity = true 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon osd allow pg remap = true 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon warn on no sortbitwise = false 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon warn on too few osds = false 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: mon_allow_pool_size_one = true 2026-02-26T03:25:08.717 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: osd default data pool replay window = 5 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: mon allow pool delete = true 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: mon cluster log file level = debug 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: debug asserts on shutdown = true 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: mon health detail to clog = false 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: mon host = 10.20.193.95 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:25:08.718 INFO:teuthology.orchestra.run.trial095.stdout: ms client mode = crc 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: ms cluster mode = crc 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: ms mon client mode = crc 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: ms mon cluster mode = crc 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: ms mon service mode = crc 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: ms service mode = crc 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: fsid = efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout:[osd] 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: osd journal size = 100 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: osd scrub load threshold = 5.0 2026-02-26T03:25:08.719 INFO:teuthology.orchestra.run.trial095.stdout: osd scrub max interval = 600 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd mclock skip benchmark = true 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd recover clone overlap = true 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd recovery max chunk = 1048576 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd debug shutdown = true 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd debug op order = true 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd debug verify stray on activate = true 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd debug trim objects = true 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.720 INFO:teuthology.orchestra.run.trial095.stdout: osd open classes on start = true 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: osd debug pg log writeout = true 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: osd map max advance = 10 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: journal zero on create = true 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: filestore apply finisher threads = 3 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: bdev debug aio = true 2026-02-26T03:25:08.721 INFO:teuthology.orchestra.run.trial095.stdout: osd debug misdirected ops = true 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug monc = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: crimson cpu num = 4 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: crimson osd obc lru size = 10 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug ms = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug osd = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore backref = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore cache = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore cleaner = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore device = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore epm = 20 2026-02-26T03:25:08.722 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore fixedkv tree = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore journal = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore lba = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore odata = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore omap = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore onode = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore t = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: debug seastore tm = 20 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: osd objectstore = seastore 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: osd shutdown pgref assert = True 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: seastore cachepin size pershard = 64M 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: seastore max concurrent transactions = 8 2026-02-26T03:25:08.723 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout:[mgr] 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: debug ms = 1 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: debug mgr = 20 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: debug mon = 20 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: debug auth = 20 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: mgr/telemetry/nag = false 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout:[mon] 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: debug ms = 1 2026-02-26T03:25:08.724 INFO:teuthology.orchestra.run.trial095.stdout: debug mon = 20 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: debug paxos = 20 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: debug auth = 20 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: mon data avail warn = 5 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: mon mgr mkfs grace = 240 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: mon osd reporter subtree level = osd 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: mon osd prime pg temp = true 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:25:08.725 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: # 1m isn't quite enough 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon min osdmap epochs = 50 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon osdmap full prune interval = 2 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon osdmap full prune min = 15 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: mon osdmap full prune txsize = 2 2026-02-26T03:25:08.726 INFO:teuthology.orchestra.run.trial095.stdout: paxos service trim min = 10 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout: osd pool default crimson = True 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout:[client] 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout: rgw cache enabled = true 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout: rgw enable ops log = true 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout: rgw enable usage log = true 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:25:08.727 INFO:teuthology.orchestra.run.trial095.stdout:[mon.a] 2026-02-26T03:25:08.729 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:25:08.729 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:25:08.818 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:25:08.818 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:08.818 DEBUG:teuthology.orchestra.run.trial095:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:25:08.835 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:08.835 DEBUG:teuthology.orchestra.run.trial095:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:25:08.891 INFO:tasks.ceph:Sending monmap to node ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:25:08.891 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:08.891 DEBUG:teuthology.orchestra.run.trial095:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:25:08.891 DEBUG:teuthology.orchestra.run.trial095:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:25:08.965 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:08.965 DEBUG:teuthology.orchestra.run.trial095:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:09.021 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:25:09.021 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:25:09.021 DEBUG:teuthology.orchestra.run.trial095:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:25:09.112 INFO:teuthology.orchestra.run.trial095.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:25:09.115 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:25:09.115 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:25:09.115 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:25:09.158 INFO:teuthology.orchestra.run.trial095.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:25:09.169 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:25:09.170 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial095.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:25:09.170 DEBUG:teuthology.orchestra.run.trial095:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:25:09.234 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:25:09.234 INFO:tasks.ceph:role: osd.0 2026-02-26T03:25:09.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:25:09.235 DEBUG:teuthology.orchestra.run.trial095:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:25:09.299 INFO:teuthology.orchestra.run.trial095.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:25:09.300 INFO:teuthology.orchestra.run.trial095.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:25:09.924 INFO:teuthology.orchestra.run.trial095.stdout:Discarding blocks...Done. 2026-02-26T03:25:09.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial095.front.sepia.ceph.com -o noatime 2026-02-26T03:25:09.926 DEBUG:teuthology.orchestra.run.trial095:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:25:09.995 DEBUG:teuthology.orchestra.run.trial095:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:25:10.065 DEBUG:teuthology.orchestra.run.trial095:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:25:10.129 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:25:10.129 INFO:tasks.ceph:role: osd.1 2026-02-26T03:25:10.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:25:10.129 DEBUG:teuthology.orchestra.run.trial095:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:25:10.194 INFO:teuthology.orchestra.run.trial095.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:25:10.195 INFO:teuthology.orchestra.run.trial095.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:25:10.195 INFO:teuthology.orchestra.run.trial095.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:25:10.754 INFO:teuthology.orchestra.run.trial095.stdout:Discarding blocks...Done. 2026-02-26T03:25:10.756 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial095.front.sepia.ceph.com -o noatime 2026-02-26T03:25:10.757 DEBUG:teuthology.orchestra.run.trial095:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:25:10.826 DEBUG:teuthology.orchestra.run.trial095:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:25:10.893 DEBUG:teuthology.orchestra.run.trial095:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:25:10.958 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:25:10.958 INFO:tasks.ceph:role: osd.2 2026-02-26T03:25:10.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:25:10.959 DEBUG:teuthology.orchestra.run.trial095:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:25:11.022 INFO:teuthology.orchestra.run.trial095.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:25:11.022 INFO:teuthology.orchestra.run.trial095.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:25:11.023 INFO:teuthology.orchestra.run.trial095.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:25:11.632 INFO:teuthology.orchestra.run.trial095.stdout:Discarding blocks...Done. 2026-02-26T03:25:11.634 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial095.front.sepia.ceph.com -o noatime 2026-02-26T03:25:11.634 DEBUG:teuthology.orchestra.run.trial095:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:25:11.702 DEBUG:teuthology.orchestra.run.trial095:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:25:11.769 DEBUG:teuthology.orchestra.run.trial095:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:11.918 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:25:11.979 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:11,978 seastar - Reactor backend: linux-aio 2026-02-26T03:25:11.988 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:11,988 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:11.989 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:11,988 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:11.991 INFO:teuthology.orchestra.run.trial095.stderr:==36929==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:12.046 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:12,044 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:25:12.046 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:12,044 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:25:12.046 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:12,045 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:25:12.046 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:12,045 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:25:12.046 INFO:teuthology.orchestra.run.trial095.stderr:DEBUG 2026-02-26 03:25:12,045 [shard 0:main] seastar - reactor::drain 2026-02-26T03:25:14.091 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:14.092 INFO:teuthology.orchestra.run.trial095.stderr:Suppressions used: 2026-02-26T03:25:14.092 INFO:teuthology.orchestra.run.trial095.stderr: count bytes template 2026-02-26T03:25:14.092 INFO:teuthology.orchestra.run.trial095.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:14.092 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:14.092 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:25:14.111 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:25:14.111 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:25:14.158 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:25:14.158 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:25:14.165 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:14,165 seastar - Reactor backend: linux-aio 2026-02-26T03:25:14.175 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:14,175 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:14.176 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:14,175 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:14.179 INFO:teuthology.orchestra.run.trial095.stderr:==36927==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:14.180 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:14,179 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:25:14.237 INFO:teuthology.orchestra.run.trial095.stderr:DEBUG 2026-02-26 03:25:14,236 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:25:14.316 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xf2e0ad 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c521cc 0x9c5a24f 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:14.316 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe1d59b 0x789462b 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:14.316 INFO:teuthology.orchestra.run.trial095.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8905b812 0xffffffff890635c7 0xffffffff88fca5f6 0xffffffff88fd4bab 0xffffffff88fd500b 0xffffffff88c910d2 0xffffffff89950451 0xffffffff89a00ba2 2026-02-26T03:25:14.317 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d8 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c521cc 0x9c5273a 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:14.461 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b357 /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xb69757 0x1090878 0xf2e0dc 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c521cc 0x9c5a24f 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:14.462 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:14.462 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb13ef 0xb69757 0x1090878 0xf2e0dc 0x9c554b6 0x9c55763 0x9d42c55 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:16.588 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:16.589 INFO:teuthology.orchestra.run.trial095.stderr:Suppressions used: 2026-02-26T03:25:16.589 INFO:teuthology.orchestra.run.trial095.stderr: count bytes template 2026-02-26T03:25:16.589 INFO:teuthology.orchestra.run.trial095.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:16.589 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:16.589 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:25:16.620 DEBUG:teuthology.orchestra.run.trial095:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:25:16.645 DEBUG:teuthology.orchestra.run.trial095:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:16.794 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:25:16.849 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:16,848 seastar - Reactor backend: linux-aio 2026-02-26T03:25:16.866 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:16,866 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:16.867 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:16,866 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:16.869 INFO:teuthology.orchestra.run.trial095.stderr:==36997==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:16.924 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:16,923 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:25:16.924 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:16,923 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:25:16.924 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:16,923 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:25:16.924 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:16,923 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:25:16.924 INFO:teuthology.orchestra.run.trial095.stderr:DEBUG 2026-02-26 03:25:16,923 [shard 0:main] seastar - reactor::drain 2026-02-26T03:25:18.970 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:18.971 INFO:teuthology.orchestra.run.trial095.stderr:Suppressions used: 2026-02-26T03:25:18.971 INFO:teuthology.orchestra.run.trial095.stderr: count bytes template 2026-02-26T03:25:18.971 INFO:teuthology.orchestra.run.trial095.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:18.971 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:18.971 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:25:18.990 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:25:18.990 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:25:19.038 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:25:19.039 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:25:19.046 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:19,045 seastar - Reactor backend: linux-aio 2026-02-26T03:25:19.055 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:19,054 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:19.055 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:19,054 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:19.059 INFO:teuthology.orchestra.run.trial095.stderr:==36995==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:19.060 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:19,059 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:25:19.118 INFO:teuthology.orchestra.run.trial095.stderr:DEBUG 2026-02-26 03:25:19,117 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:25:19.196 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xcb061d 0x11253a2 0xf2e0d1 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d368ca 0x9d3851c 0x9d38b76 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:19.197 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xe0c0ae 0x9d26268 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:19.197 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:19.345 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xb6e473 0x9d479cd 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:19.345 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12985d /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d38e4d 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:19.345 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xa344 /lib64/libasan.so.8+0xc478 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba35e4 0x9ba325d 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9d45991 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:21.475 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:21.475 INFO:teuthology.orchestra.run.trial095.stderr:Suppressions used: 2026-02-26T03:25:21.475 INFO:teuthology.orchestra.run.trial095.stderr: count bytes template 2026-02-26T03:25:21.475 INFO:teuthology.orchestra.run.trial095.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:21.476 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:21.476 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:25:21.506 DEBUG:teuthology.orchestra.run.trial095:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:25:21.531 DEBUG:teuthology.orchestra.run.trial095:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:21.680 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:25:21.738 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:21,737 seastar - Reactor backend: linux-aio 2026-02-26T03:25:21.756 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:21,755 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:21.756 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:21,755 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:21.758 INFO:teuthology.orchestra.run.trial095.stderr:==37063==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:21.813 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:21,812 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:25:21.813 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:21,812 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:25:21.813 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:21,812 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:25:21.813 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:21,812 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:25:21.814 INFO:teuthology.orchestra.run.trial095.stderr:DEBUG 2026-02-26 03:25:21,812 [shard 0:main] seastar - reactor::drain 2026-02-26T03:25:23.862 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:23.863 INFO:teuthology.orchestra.run.trial095.stderr:Suppressions used: 2026-02-26T03:25:23.863 INFO:teuthology.orchestra.run.trial095.stderr: count bytes template 2026-02-26T03:25:23.863 INFO:teuthology.orchestra.run.trial095.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:23.863 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:23.863 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:25:23.882 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:25:23.882 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:25:23.930 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:25:23.930 INFO:teuthology.orchestra.run.trial095.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:25:23.937 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:23,936 seastar - Reactor backend: linux-aio 2026-02-26T03:25:23.945 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:23,945 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:23.946 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:23,945 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:23.950 INFO:teuthology.orchestra.run.trial095.stderr:==37061==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:23.950 INFO:teuthology.orchestra.run.trial095.stderr:INFO 2026-02-26 03:25:23,949 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:25:24.007 INFO:teuthology.orchestra.run.trial095.stderr:DEBUG 2026-02-26 03:25:24,006 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:25:24.085 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12985d /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c5477a 0x9c550b9 0x9d3f001 0x9d496e7 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:24.086 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15192 0x316f517 0x9c511ba 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:24.086 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x9c50af8 0x9c50f59 0x9c511cc 0x9c52ef3 0x9c53440 0x9c5363f 0x9d38961 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:24.231 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65bc /lib64/libasan.so.8+0x157c7 0x9c51220 0x9c521cc 0x9c5a24f 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7beb0f2 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:24.231 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9d0c53e 0x9d0c66c 0x9d3ea77 0x9d3f21a 0x9d496e7 0x78947dd 0x7beb047 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:24.231 INFO:teuthology.orchestra.run.trial095.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88c75c69 0xffffffff88e20157 0xffffffff88e203b0 0xffffffff88c9525e 0xffffffff88fe7269 0xffffffff88fca7d5 0xffffffff88fd4bab 0xffffffff88fd500b 0xffffffff88c910d2 0xffffffff89950451 0xffffffff89a00ba2 2026-02-26T03:25:24.231 INFO:teuthology.orchestra.run.trial095.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x10906d3 0xf2e0dc 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c521cc 0x9c5a24f 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:26.357 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:26.358 INFO:teuthology.orchestra.run.trial095.stderr:Suppressions used: 2026-02-26T03:25:26.358 INFO:teuthology.orchestra.run.trial095.stderr: count bytes template 2026-02-26T03:25:26.358 INFO:teuthology.orchestra.run.trial095.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:26.358 INFO:teuthology.orchestra.run.trial095.stderr:----------------------------------------------------- 2026-02-26T03:25:26.358 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:25:26.390 DEBUG:teuthology.orchestra.run.trial095:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:25:26.416 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:25:26.417 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:26.417 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:25:26.481 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:26.481 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:25:26.545 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:26.545 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:25:26.608 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:26.608 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:25:26.671 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:26.671 DEBUG:teuthology.orchestra.run.trial095:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:25:26.727 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:25:26.727 DEBUG:teuthology.orchestra.run.trial095:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:25:26.789 INFO:teuthology.orchestra.run.trial095.stdout:[mgr.x] 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout: key = AQAVvZ9pGuKkBhAA2lHtD8ma+RH7xAJaGpULwQ== 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout:[osd.0] 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout: key = AQAavZ9pqdRTDhAAwS5Tnk3LlRYENGxa+09QmA== 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout:[osd.1] 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout: key = AQAfvZ9pAsE2BxAA15LS/AE4MrBSwo4S4q/kMQ== 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout:[osd.2] 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout: key = AQAkvZ9po4+aABAAfE2hlfrgSyXoU2vF1LyI0A== 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout:[client.0] 2026-02-26T03:25:26.790 INFO:teuthology.orchestra.run.trial095.stdout: key = AQAVvZ9pkctkCRAABNDFSdEWOBTX589nOp1hRQ== 2026-02-26T03:25:26.792 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:25:26.880 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:25:26.930 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:25:26.978 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:25:27.027 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:25:27.075 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:25:27.076 DEBUG:teuthology.orchestra.run.trial095:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:25:27.099 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:25:27.186 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.184+0000 7f38c213bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.187 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.184+0000 7f38c213bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.198 DEBUG:teuthology.orchestra.run.trial095:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:25:27.223 DEBUG:teuthology.orchestra.run.trial095:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:27.279 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:25:27.280 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:25:27.280 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:25:27.321 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:25:27.321 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:25:27.322 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:25:27.322 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:25:27.324 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:25:27.324 DEBUG:tasks.ceph:set 0 configs 2026-02-26T03:25:27.324 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph config dump 2026-02-26T03:25:27.385 INFO:tasks.ceph.mon.a.trial095.stderr:2026-02-26T03:25:27.383+0000 7f3c1b681c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.386 INFO:tasks.ceph.mon.a.trial095.stderr:2026-02-26T03:25:27.383+0000 7f3c1b681c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.386 INFO:tasks.ceph.mon.a.trial095.stderr:2026-02-26T03:25:27.383+0000 7f3c1b681c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.389 INFO:tasks.ceph.mgr.x.trial095.stderr:2026-02-26T03:25:27.387+0000 7fd192a2cfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.400 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.395+0000 7f6114a11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.403 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.401+0000 7f6114a11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.436 INFO:teuthology.orchestra.run.trial095.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-26T03:25:27.444 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:25:27.444 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:25:27.517 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.512+0000 7f044750b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.520 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.518+0000 7f044750b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.554 INFO:teuthology.orchestra.run.trial095.stderr:adjusted tunables profile to default 2026-02-26T03:25:27.563 INFO:tasks.ceph:check_enable_crimson: True 2026-02-26T03:25:27.563 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-26T03:25:27.563 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-26T03:25:27.593 INFO:tasks.ceph.mgr.x.trial095.stderr:2026-02-26T03:25:27.591+0000 7fd192a2cfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.593 INFO:tasks.ceph.mgr.x.trial095.stderr:2026-02-26T03:25:27.591+0000 7fd192a2cfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.637 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.632+0000 7f4c04a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.639 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:27.637+0000 7f4c04a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:27.680 INFO:tasks.ceph.mon.a.trial095.stderr:2026-02-26T03:25:27.678+0000 7f3c12df1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-26T03:25:27.680 INFO:tasks.ceph.mon.a.trial095.stderr:Please be aware that this feature is experimental, untested, 2026-02-26T03:25:27.680 INFO:tasks.ceph.mon.a.trial095.stderr:unsupported, and may result in data corruption, data loss, 2026-02-26T03:25:27.680 INFO:tasks.ceph.mon.a.trial095.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-26T03:25:27.680 INFO:tasks.ceph.mon.a.trial095.stderr:feature with important data. 2026-02-26T03:25:27.680 INFO:tasks.ceph.mon.a.trial095.stderr: 2026-02-26T03:25:28.435 INFO:tasks.ceph.mgr.x.trial095.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:25:28.435 INFO:tasks.ceph.mgr.x.trial095.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:25:28.435 INFO:tasks.ceph.mgr.x.trial095.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:25:28.565 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:25:28.565 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:28.566 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:25:28.629 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:28.629 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:25:28.694 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:25:28.694 DEBUG:teuthology.orchestra.run.trial095:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:25:28.759 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph osd new c24bac03-1167-4daf-8c9b-92e20facb087 0 2026-02-26T03:25:28.874 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:28.868+0000 7fdad678c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:28.876 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:28.874+0000 7fdad678c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:28.914 INFO:teuthology.orchestra.run.trial095.stdout:0 2026-02-26T03:25:28.923 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph osd new 0033ac36-5e4b-47ed-b8d7-fe23edfa432d 1 2026-02-26T03:25:28.996 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:28.991+0000 7f0972ab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:28.998 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:28.996+0000 7f0972ab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.036 INFO:teuthology.orchestra.run.trial095.stdout:1 2026-02-26T03:25:29.044 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph osd new a085a950-c9a9-46a1-a73d-897375a15a1f 2 2026-02-26T03:25:29.116 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:29.111+0000 7f4a9a2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.118 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:29.116+0000 7f4a9a2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.155 INFO:teuthology.orchestra.run.trial095.stdout:2 2026-02-26T03:25:29.163 DEBUG:teuthology.orchestra.run.trial095:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-26T03:25:29.191 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:25:29.192 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-26T03:25:29.235 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:25:29.235 DEBUG:teuthology.orchestra.run.trial095:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-26T03:25:29.264 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:25:29.264 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-26T03:25:29.306 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:25:29.306 DEBUG:teuthology.orchestra.run.trial095:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-26T03:25:29.332 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:25:29.332 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-26T03:25:29.374 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:25:29.374 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:29.450 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:29.445+0000 7f6efe435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.452 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:29.450+0000 7f6efe435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.486 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:29.486 INFO:teuthology.orchestra.run.trial095.stdout:{"epoch":6,"fsid":"efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e","created":"2026-02-26T03:25:27.396677+0000","modified":"2026-02-26T03:25:29.154553+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:25:29.154553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c24bac03-1167-4daf-8c9b-92e20facb087","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0033ac36-5e4b-47ed-b8d7-fe23edfa432d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a085a950-c9a9-46a1-a73d-897375a15a1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:29.495 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:25:29.495 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:25:29.797 DEBUG:teuthology.orchestra.run.trial095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:29.865 INFO:teuthology.misc.health.trial095.stderr:2026-02-26T03:25:29.861+0000 7f220d65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.867 INFO:teuthology.misc.health.trial095.stderr:2026-02-26T03:25:29.865+0000 7f220d65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.900 INFO:teuthology.misc.health.trial095.stdout: 2026-02-26T03:25:29.900 INFO:teuthology.misc.health.trial095.stdout:{"epoch":6,"fsid":"efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e","created":"2026-02-26T03:25:27.396677+0000","modified":"2026-02-26T03:25:29.154553+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:25:29.154553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c24bac03-1167-4daf-8c9b-92e20facb087","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0033ac36-5e4b-47ed-b8d7-fe23edfa432d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a085a950-c9a9-46a1-a73d-897375a15a1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:29.906 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:25:30.841 INFO:tasks.ceph.mgr.x.trial095.stderr:2026-02-26T03:25:30.839+0000 7fd0ad604640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:25:36.208 DEBUG:teuthology.orchestra.run.trial095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:36.276 INFO:teuthology.misc.health.trial095.stderr:2026-02-26T03:25:36.272+0000 7fcea201a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:36.278 INFO:teuthology.misc.health.trial095.stderr:2026-02-26T03:25:36.276+0000 7fcea201a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:36.390 INFO:teuthology.misc.health.trial095.stdout: 2026-02-26T03:25:36.390 INFO:teuthology.misc.health.trial095.stdout:{"epoch":7,"fsid":"efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e","created":"2026-02-26T03:25:27.396677+0000","modified":"2026-02-26T03:25:35.683752+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:25:29.154553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c24bac03-1167-4daf-8c9b-92e20facb087","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0033ac36-5e4b-47ed-b8d7-fe23edfa432d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a085a950-c9a9-46a1-a73d-897375a15a1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:36.397 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:25:42.700 DEBUG:teuthology.orchestra.run.trial095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:42.769 INFO:teuthology.misc.health.trial095.stderr:2026-02-26T03:25:42.765+0000 7f9865c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:42.771 INFO:teuthology.misc.health.trial095.stderr:2026-02-26T03:25:42.769+0000 7f9865c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:42.882 INFO:teuthology.misc.health.trial095.stdout: 2026-02-26T03:25:42.882 INFO:teuthology.misc.health.trial095.stdout:{"epoch":12,"fsid":"efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e","created":"2026-02-26T03:25:27.396677+0000","modified":"2026-02-26T03:25:40.708357+0000","last_up_change":"2026-02-26T03:25:38.697555+0000","last_in_change":"2026-02-26T03:25:29.154553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:37.853178+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c24bac03-1167-4daf-8c9b-92e20facb087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6804","nonce":3851961937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6805","nonce":3851961937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6806","nonce":3851961937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6809","nonce":3851961937}]},"public_addr":"10.20.193.95:6804/3851961937","cluster_addr":"10.20.193.95:6805/3851961937","heartbeat_back_addr":"10.20.193.95:6806/3851961937","heartbeat_front_addr":"10.20.193.95:6809/3851961937","state":["exists","up"]},{"osd":1,"uuid":"0033ac36-5e4b-47ed-b8d7-fe23edfa432d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6802","nonce":1661387041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6803","nonce":1661387041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6810","nonce":1661387041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6811","nonce":1661387041}]},"public_addr":"10.20.193.95:6802/1661387041","cluster_addr":"10.20.193.95:6803/1661387041","heartbeat_back_addr":"10.20.193.95:6810/1661387041","heartbeat_front_addr":"10.20.193.95:6811/1661387041","state":["exists","up"]},{"osd":2,"uuid":"a085a950-c9a9-46a1-a73d-897375a15a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6808","nonce":2801899040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6807","nonce":2801899040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6814","nonce":2801899040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6813","nonce":2801899040}]},"public_addr":"10.20.193.95:6808/2801899040","cluster_addr":"10.20.193.95:6807/2801899040","heartbeat_back_addr":"10.20.193.95:6814/2801899040","heartbeat_front_addr":"10.20.193.95:6813/2801899040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:42.891 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:25:42.891 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:25:42.891 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:25:42.966 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:42.960+0000 7ff6352cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:42.968 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:42.966+0000 7ff6352cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:43.718 INFO:teuthology.orchestra.run.trial095.stderr:pool 'rbd' created 2026-02-26T03:25:43.731 DEBUG:teuthology.orchestra.run.trial095:> rbd --cluster ceph pool init rbd 2026-02-26T03:25:43.774 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:43.772+0000 7fc90e8df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:43.774 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:43.772+0000 7fc90e8df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:43.776 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:43.775+0000 7fc90e8df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:46.753 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:25:46.753 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-26T03:25:46.754 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:25:46.870 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:46.866+0000 7f02d928b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:46.872 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:46.871+0000 7f02d928b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:46.983 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:46.993 INFO:teuthology.orchestra.run.trial095.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-26T03:25:46.993 INFO:tasks.ceph_manager:config epoch is 1 2026-02-26T03:25:46.993 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:25:46.993 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:25:46.994 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:25:47.070 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.067+0000 7f1c3365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.072 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.071+0000 7f1c3365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.216 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:47.225 INFO:teuthology.orchestra.run.trial095.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6800","nonce":1368439407},{"type":"v1","addr":"10.20.193.95:6801","nonce":1368439407}]},"active_addr":"10.20.193.95:6801/1368439407","active_change":"2026-02-26T03:25:29.834868+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.95:0","nonce":2542853550}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.95:0","nonce":4094011362}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.95:0","nonce":2114411089}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.95:0","nonce":2784805929}]}]} 2026-02-26T03:25:47.229 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:25:47.229 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:25:47.229 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:47.304 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.301+0000 7f2238562640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.306 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.305+0000 7f2238562640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.417 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:47.418 INFO:teuthology.orchestra.run.trial095.stdout:{"epoch":16,"fsid":"efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e","created":"2026-02-26T03:25:27.396677+0000","modified":"2026-02-26T03:25:46.725108+0000","last_up_change":"2026-02-26T03:25:38.697555+0000","last_in_change":"2026-02-26T03:25:29.154553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:37.853178+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:25:43.083700+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c24bac03-1167-4daf-8c9b-92e20facb087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6804","nonce":3851961937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6805","nonce":3851961937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6806","nonce":3851961937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6809","nonce":3851961937}]},"public_addr":"10.20.193.95:6804/3851961937","cluster_addr":"10.20.193.95:6805/3851961937","heartbeat_back_addr":"10.20.193.95:6806/3851961937","heartbeat_front_addr":"10.20.193.95:6809/3851961937","state":["exists","up"]},{"osd":1,"uuid":"0033ac36-5e4b-47ed-b8d7-fe23edfa432d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6802","nonce":1661387041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6803","nonce":1661387041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6810","nonce":1661387041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6811","nonce":1661387041}]},"public_addr":"10.20.193.95:6802/1661387041","cluster_addr":"10.20.193.95:6803/1661387041","heartbeat_back_addr":"10.20.193.95:6810/1661387041","heartbeat_front_addr":"10.20.193.95:6811/1661387041","state":["exists","up"]},{"osd":2,"uuid":"a085a950-c9a9-46a1-a73d-897375a15a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6808","nonce":2801899040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6807","nonce":2801899040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6814","nonce":2801899040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6813","nonce":2801899040}]},"public_addr":"10.20.193.95:6808/2801899040","cluster_addr":"10.20.193.95:6807/2801899040","heartbeat_back_addr":"10.20.193.95:6814/2801899040","heartbeat_front_addr":"10.20.193.95:6813/2801899040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:47.427 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:25:47.427 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:47.503 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.500+0000 7fafd883d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.505 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.504+0000 7fafd75b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.617 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:47.617 INFO:teuthology.orchestra.run.trial095.stdout:{"epoch":16,"fsid":"efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e","created":"2026-02-26T03:25:27.396677+0000","modified":"2026-02-26T03:25:46.725108+0000","last_up_change":"2026-02-26T03:25:38.697555+0000","last_in_change":"2026-02-26T03:25:29.154553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:37.853178+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:25:43.083700+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c24bac03-1167-4daf-8c9b-92e20facb087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6804","nonce":3851961937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6805","nonce":3851961937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6806","nonce":3851961937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6809","nonce":3851961937}]},"public_addr":"10.20.193.95:6804/3851961937","cluster_addr":"10.20.193.95:6805/3851961937","heartbeat_back_addr":"10.20.193.95:6806/3851961937","heartbeat_front_addr":"10.20.193.95:6809/3851961937","state":["exists","up"]},{"osd":1,"uuid":"0033ac36-5e4b-47ed-b8d7-fe23edfa432d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6802","nonce":1661387041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6803","nonce":1661387041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6810","nonce":1661387041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6811","nonce":1661387041}]},"public_addr":"10.20.193.95:6802/1661387041","cluster_addr":"10.20.193.95:6803/1661387041","heartbeat_back_addr":"10.20.193.95:6810/1661387041","heartbeat_front_addr":"10.20.193.95:6811/1661387041","state":["exists","up"]},{"osd":2,"uuid":"a085a950-c9a9-46a1-a73d-897375a15a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6808","nonce":2801899040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6807","nonce":2801899040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6814","nonce":2801899040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6813","nonce":2801899040}]},"public_addr":"10.20.193.95:6808/2801899040","cluster_addr":"10.20.193.95:6807/2801899040","heartbeat_back_addr":"10.20.193.95:6814/2801899040","heartbeat_front_addr":"10.20.193.95:6813/2801899040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:47.627 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:25:47.628 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:25:47.628 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:25:47.710 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.705+0000 7f80282cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.710 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.705+0000 7f6ed2851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.710 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.706+0000 7fc0dbacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.712 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.711+0000 7f6ed2851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.712 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.711+0000 7f80282cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.713 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.712+0000 7fc0dbacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.729 INFO:teuthology.orchestra.run.trial095.stdout:38654705674 2026-02-26T03:25:47.729 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:25:47.730 INFO:teuthology.orchestra.run.trial095.stdout:38654705674 2026-02-26T03:25:47.731 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:25:47.732 INFO:teuthology.orchestra.run.trial095.stdout:42949672969 2026-02-26T03:25:47.732 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:25:47.851 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.847+0000 7f6d4f10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.852 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.847+0000 7fddb04b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.852 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.847+0000 7f2d141a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.853 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.852+0000 7f6d4f10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.854 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.853+0000 7fddb04b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.854 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:47.853+0000 7f2d141a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:47.967 INFO:teuthology.orchestra.run.trial095.stdout:38654705674 2026-02-26T03:25:47.970 INFO:teuthology.orchestra.run.trial095.stdout:42949672969 2026-02-26T03:25:47.973 INFO:teuthology.orchestra.run.trial095.stdout:38654705674 2026-02-26T03:25:47.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-26T03:25:47.978 DEBUG:teuthology.parallel:result is None 2026-02-26T03:25:47.981 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-26T03:25:47.981 DEBUG:teuthology.parallel:result is None 2026-02-26T03:25:47.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2026-02-26T03:25:47.985 DEBUG:teuthology.parallel:result is None 2026-02-26T03:25:47.985 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:25:47.986 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:25:48.100 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:48.096+0000 7f50d278c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:48.102 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:48.101+0000 7f50d278c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:48.212 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:48.212 INFO:teuthology.orchestra.run.trial095.stderr:dumped all 2026-02-26T03:25:48.221 INFO:teuthology.orchestra.run.trial095.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-26T03:25:47.838975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917004,"kb_used_data":62917004,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369348,"statfs":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.121111"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.736306+0000","last_change":"2026-02-26T03:25:46.738834+0000","last_active":"2026-02-26T03:25:46.736306+0000","last_peered":"2026-02-26T03:25:46.736306+0000","last_clean":"2026-02-26T03:25:46.736306+0000","last_became_active":"2026-02-26T03:25:44.759336+0000","last_became_peered":"2026-02-26T03:25:44.759336+0000","last_unstale":"2026-02-26T03:25:46.736306+0000","last_undegraded":"2026-02-26T03:25:46.736306+0000","last_fullsized":"2026-02-26T03:25:46.736306+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737523+0000","last_change":"2026-02-26T03:25:46.740678+0000","last_active":"2026-02-26T03:25:46.737523+0000","last_peered":"2026-02-26T03:25:46.737523+0000","last_clean":"2026-02-26T03:25:46.737523+0000","last_became_active":"2026-02-26T03:25:44.758719+0000","last_became_peered":"2026-02-26T03:25:44.758719+0000","last_unstale":"2026-02-26T03:25:46.737523+0000","last_undegraded":"2026-02-26T03:25:46.737523+0000","last_fullsized":"2026-02-26T03:25:46.737523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737342+0000","last_change":"2026-02-26T03:25:46.741334+0000","last_active":"2026-02-26T03:25:46.737342+0000","last_peered":"2026-02-26T03:25:46.737342+0000","last_clean":"2026-02-26T03:25:46.737342+0000","last_became_active":"2026-02-26T03:25:44.769736+0000","last_became_peered":"2026-02-26T03:25:44.769736+0000","last_unstale":"2026-02-26T03:25:46.737342+0000","last_undegraded":"2026-02-26T03:25:46.737342+0000","last_fullsized":"2026-02-26T03:25:46.737342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737471+0000","last_change":"2026-02-26T03:25:46.739713+0000","last_active":"2026-02-26T03:25:46.737471+0000","last_peered":"2026-02-26T03:25:46.737471+0000","last_clean":"2026-02-26T03:25:46.737471+0000","last_became_active":"2026-02-26T03:25:44.764783+0000","last_became_peered":"2026-02-26T03:25:44.764783+0000","last_unstale":"2026-02-26T03:25:46.737471+0000","last_undegraded":"2026-02-26T03:25:46.737471+0000","last_fullsized":"2026-02-26T03:25:46.737471+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.740773+0000","last_change":"2026-02-26T03:25:46.741963+0000","last_active":"2026-02-26T03:25:46.740773+0000","last_peered":"2026-02-26T03:25:46.740773+0000","last_clean":"2026-02-26T03:25:46.740773+0000","last_became_active":"2026-02-26T03:25:44.770887+0000","last_became_peered":"2026-02-26T03:25:44.770887+0000","last_unstale":"2026-02-26T03:25:46.740773+0000","last_undegraded":"2026-02-26T03:25:46.740773+0000","last_fullsized":"2026-02-26T03:25:46.740773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737246+0000","last_change":"2026-02-26T03:25:46.739851+0000","last_active":"2026-02-26T03:25:46.737246+0000","last_peered":"2026-02-26T03:25:46.737246+0000","last_clean":"2026-02-26T03:25:46.737246+0000","last_became_active":"2026-02-26T03:25:44.780653+0000","last_became_peered":"2026-02-26T03:25:44.780653+0000","last_unstale":"2026-02-26T03:25:46.737246+0000","last_undegraded":"2026-02-26T03:25:46.737246+0000","last_fullsized":"2026-02-26T03:25:46.737246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.734977+0000","last_change":"2026-02-26T03:25:46.737785+0000","last_active":"2026-02-26T03:25:46.734977+0000","last_peered":"2026-02-26T03:25:46.734977+0000","last_clean":"2026-02-26T03:25:46.734977+0000","last_became_active":"2026-02-26T03:25:44.775206+0000","last_became_peered":"2026-02-26T03:25:44.775206+0000","last_unstale":"2026-02-26T03:25:46.734977+0000","last_undegraded":"2026-02-26T03:25:46.734977+0000","last_fullsized":"2026-02-26T03:25:46.734977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737673+0000","last_change":"2026-02-26T03:25:46.741695+0000","last_active":"2026-02-26T03:25:46.737673+0000","last_peered":"2026-02-26T03:25:46.737673+0000","last_clean":"2026-02-26T03:25:46.737673+0000","last_became_active":"2026-02-26T03:25:44.758142+0000","last_became_peered":"2026-02-26T03:25:44.758142+0000","last_unstale":"2026-02-26T03:25:46.737673+0000","last_undegraded":"2026-02-26T03:25:46.737673+0000","last_fullsized":"2026-02-26T03:25:46.737673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":90,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.735032+0000","last_change":"2026-02-26T03:25:39.742660+0000","last_active":"2026-02-26T03:25:46.735032+0000","last_peered":"2026-02-26T03:25:46.735032+0000","last_clean":"2026-02-26T03:25:46.735032+0000","last_became_active":"2026-02-26T03:25:39.733389+0000","last_became_peered":"2026-02-26T03:25:39.733389+0000","last_unstale":"2026-02-26T03:25:46.735032+0000","last_undegraded":"2026-02-26T03:25:46.735032+0000","last_fullsized":"2026-02-26T03:25:46.735032+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_clean_scrub_stamp":"2026-02-26T03:25:38.697555+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972620,"kb_used_data":20972620,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456164,"statfs":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972572,"kb_used_data":20972572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456212,"statfs":{"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:25:48.222 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:25:48.298 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:48.296+0000 7efef838c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:48.301 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:48.300+0000 7efef838c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:48.411 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:48.412 INFO:teuthology.orchestra.run.trial095.stderr:dumped all 2026-02-26T03:25:48.420 INFO:teuthology.orchestra.run.trial095.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-26T03:25:47.838975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917004,"kb_used_data":62917004,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369348,"statfs":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.121111"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.736306+0000","last_change":"2026-02-26T03:25:46.738834+0000","last_active":"2026-02-26T03:25:46.736306+0000","last_peered":"2026-02-26T03:25:46.736306+0000","last_clean":"2026-02-26T03:25:46.736306+0000","last_became_active":"2026-02-26T03:25:44.759336+0000","last_became_peered":"2026-02-26T03:25:44.759336+0000","last_unstale":"2026-02-26T03:25:46.736306+0000","last_undegraded":"2026-02-26T03:25:46.736306+0000","last_fullsized":"2026-02-26T03:25:46.736306+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737523+0000","last_change":"2026-02-26T03:25:46.740678+0000","last_active":"2026-02-26T03:25:46.737523+0000","last_peered":"2026-02-26T03:25:46.737523+0000","last_clean":"2026-02-26T03:25:46.737523+0000","last_became_active":"2026-02-26T03:25:44.758719+0000","last_became_peered":"2026-02-26T03:25:44.758719+0000","last_unstale":"2026-02-26T03:25:46.737523+0000","last_undegraded":"2026-02-26T03:25:46.737523+0000","last_fullsized":"2026-02-26T03:25:46.737523+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737342+0000","last_change":"2026-02-26T03:25:46.741334+0000","last_active":"2026-02-26T03:25:46.737342+0000","last_peered":"2026-02-26T03:25:46.737342+0000","last_clean":"2026-02-26T03:25:46.737342+0000","last_became_active":"2026-02-26T03:25:44.769736+0000","last_became_peered":"2026-02-26T03:25:44.769736+0000","last_unstale":"2026-02-26T03:25:46.737342+0000","last_undegraded":"2026-02-26T03:25:46.737342+0000","last_fullsized":"2026-02-26T03:25:46.737342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737471+0000","last_change":"2026-02-26T03:25:46.739713+0000","last_active":"2026-02-26T03:25:46.737471+0000","last_peered":"2026-02-26T03:25:46.737471+0000","last_clean":"2026-02-26T03:25:46.737471+0000","last_became_active":"2026-02-26T03:25:44.764783+0000","last_became_peered":"2026-02-26T03:25:44.764783+0000","last_unstale":"2026-02-26T03:25:46.737471+0000","last_undegraded":"2026-02-26T03:25:46.737471+0000","last_fullsized":"2026-02-26T03:25:46.737471+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.740773+0000","last_change":"2026-02-26T03:25:46.741963+0000","last_active":"2026-02-26T03:25:46.740773+0000","last_peered":"2026-02-26T03:25:46.740773+0000","last_clean":"2026-02-26T03:25:46.740773+0000","last_became_active":"2026-02-26T03:25:44.770887+0000","last_became_peered":"2026-02-26T03:25:44.770887+0000","last_unstale":"2026-02-26T03:25:46.740773+0000","last_undegraded":"2026-02-26T03:25:46.740773+0000","last_fullsized":"2026-02-26T03:25:46.740773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737246+0000","last_change":"2026-02-26T03:25:46.739851+0000","last_active":"2026-02-26T03:25:46.737246+0000","last_peered":"2026-02-26T03:25:46.737246+0000","last_clean":"2026-02-26T03:25:46.737246+0000","last_became_active":"2026-02-26T03:25:44.780653+0000","last_became_peered":"2026-02-26T03:25:44.780653+0000","last_unstale":"2026-02-26T03:25:46.737246+0000","last_undegraded":"2026-02-26T03:25:46.737246+0000","last_fullsized":"2026-02-26T03:25:46.737246+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.734977+0000","last_change":"2026-02-26T03:25:46.737785+0000","last_active":"2026-02-26T03:25:46.734977+0000","last_peered":"2026-02-26T03:25:46.734977+0000","last_clean":"2026-02-26T03:25:46.734977+0000","last_became_active":"2026-02-26T03:25:44.775206+0000","last_became_peered":"2026-02-26T03:25:44.775206+0000","last_unstale":"2026-02-26T03:25:46.734977+0000","last_undegraded":"2026-02-26T03:25:46.734977+0000","last_fullsized":"2026-02-26T03:25:46.734977+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.737673+0000","last_change":"2026-02-26T03:25:46.741695+0000","last_active":"2026-02-26T03:25:46.737673+0000","last_peered":"2026-02-26T03:25:46.737673+0000","last_clean":"2026-02-26T03:25:46.737673+0000","last_became_active":"2026-02-26T03:25:44.758142+0000","last_became_peered":"2026-02-26T03:25:44.758142+0000","last_unstale":"2026-02-26T03:25:46.737673+0000","last_undegraded":"2026-02-26T03:25:46.737673+0000","last_fullsized":"2026-02-26T03:25:46.737673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":90,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-26T03:25:46.735032+0000","last_change":"2026-02-26T03:25:39.742660+0000","last_active":"2026-02-26T03:25:46.735032+0000","last_peered":"2026-02-26T03:25:46.735032+0000","last_clean":"2026-02-26T03:25:46.735032+0000","last_became_active":"2026-02-26T03:25:39.733389+0000","last_became_peered":"2026-02-26T03:25:39.733389+0000","last_unstale":"2026-02-26T03:25:46.735032+0000","last_undegraded":"2026-02-26T03:25:46.735032+0000","last_fullsized":"2026-02-26T03:25:46.735032+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_clean_scrub_stamp":"2026-02-26T03:25:38.697555+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972620,"kb_used_data":20972620,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456164,"statfs":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972572,"kb_used_data":20972572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456212,"statfs":{"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:25:48.422 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:25:48.422 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:25:48.422 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:25:48.422 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:25:48.496 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:48.494+0000 7fd0f2ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:48.499 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:25:48.498+0000 7fd0f2ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:48.637 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:25:48.637 INFO:teuthology.orchestra.run.trial095.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:25:48.647 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:25:48.647 INFO:teuthology.run_tasks:Running task workunit... 2026-02-26T03:25:48.667 INFO:tasks.workunit:Pulling workunits from ref 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 2026-02-26T03:25:48.668 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-26T03:25:48.668 DEBUG:teuthology.orchestra.run.trial095:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:25:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:25:48.685 INFO:teuthology.orchestra.run.trial095.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-26T03:25:48.686 DEBUG:teuthology.orchestra.run.trial095:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:25:48.742 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-26T03:25:48.743 DEBUG:teuthology.orchestra.run.trial095:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-26T03:25:48.799 INFO:tasks.workunit:timeout=3h 2026-02-26T03:25:48.799 INFO:tasks.workunit:cleanup=True 2026-02-26T03:25:48.799 DEBUG:teuthology.orchestra.run.trial095:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 2026-02-26T03:25:48.862 INFO:tasks.workunit.client.0.trial095.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-26T03:26:12.034 INFO:tasks.workunit.client.0.trial095.stderr:Note: switching to '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'. 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr: 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr:state without impacting any branches by switching back to a branch. 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr: 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr: 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr: git switch -c 2026-02-26T03:26:12.035 INFO:tasks.workunit.client.0.trial095.stderr: 2026-02-26T03:26:12.036 INFO:tasks.workunit.client.0.trial095.stderr:Or undo this operation with: 2026-02-26T03:26:12.036 INFO:tasks.workunit.client.0.trial095.stderr: 2026-02-26T03:26:12.036 INFO:tasks.workunit.client.0.trial095.stderr: git switch - 2026-02-26T03:26:12.036 INFO:tasks.workunit.client.0.trial095.stderr: 2026-02-26T03:26:12.036 INFO:tasks.workunit.client.0.trial095.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-26T03:26:12.036 INFO:tasks.workunit.client.0.trial095.stderr: 2026-02-26T03:26:12.036 INFO:tasks.workunit.client.0.trial095.stderr:HEAD is now at 1f87b69b1a8 Merge pull request #67522 from rhcs-dashboard/notification-banner-simple-mode 2026-02-26T03:26:12.037 DEBUG:teuthology.orchestra.run.trial095:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-26T03:26:12.093 INFO:tasks.workunit.client.0.trial095.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-26T03:26:12.095 INFO:tasks.workunit.client.0.trial095.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T03:26:12.095 INFO:tasks.workunit.client.0.trial095.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-26T03:26:12.182 INFO:tasks.workunit.client.0.trial095.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-26T03:26:12.211 INFO:tasks.workunit.client.0.trial095.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-26T03:26:12.234 INFO:tasks.workunit.client.0.trial095.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T03:26:12.236 INFO:tasks.workunit.client.0.trial095.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T03:26:12.236 INFO:tasks.workunit.client.0.trial095.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-26T03:26:12.260 INFO:tasks.workunit.client.0.trial095.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T03:26:12.264 DEBUG:teuthology.orchestra.run.trial095:> set -ex 2026-02-26T03:26:12.264 DEBUG:teuthology.orchestra.run.trial095:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-26T03:26:12.322 INFO:tasks.workunit:Running workunits matching rbd/test_librbd_python.sh on client.0... 2026-02-26T03:26:12.325 INFO:tasks.workunit:Running workunit rbd/test_librbd_python.sh... 2026-02-26T03:26:12.325 DEBUG:teuthology.orchestra.run.trial095:workunit test rbd/test_librbd_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2026-02-26T03:26:12.381 INFO:tasks.workunit.client.0.trial095.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2026-02-26T03:26:12.382 INFO:tasks.workunit.client.0.trial095.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind 2026-02-26T03:26:12.382 INFO:tasks.workunit.client.0.trial095.stderr:+ '[' -n '' ']' 2026-02-26T03:26:12.382 INFO:tasks.workunit.client.0.trial095.stderr:+ python3 -m pytest -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind/test_rbd.py 2026-02-26T03:26:12.457 INFO:tasks.workunit.client.0.trial095.stdout:============================= test session starts ============================== 2026-02-26T03:26:12.457 INFO:tasks.workunit.client.0.trial095.stdout:platform linux -- Python 3.9.25, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 2026-02-26T03:26:12.457 INFO:tasks.workunit.client.0.trial095.stdout:cachedir: .pytest_cache 2026-02-26T03:26:12.457 INFO:tasks.workunit.client.0.trial095.stdout:rootdir: /home/ubuntu/cephtest/clone.client.0/src/test/pybind, configfile: pytest.ini 2026-02-26T03:26:12.751 INFO:tasks.workunit.client.0.trial095.stdout:collecting ... collected 196 items 2026-02-26T03:26:12.751 INFO:tasks.workunit.client.0.trial095.stdout: 2026-02-26T03:26:15.959 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_version PASSED [ 0%] 2026-02-26T03:26:16.457 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create PASSED [ 1%] 2026-02-26T03:26:20.923 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_defaults PASSED [ 1%] 2026-02-26T03:26:21.594 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_context_manager PASSED [ 2%] 2026-02-26T03:26:23.951 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_read_only PASSED [ 2%] 2026-02-26T03:26:25.087 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_dne PASSED [ 3%] 2026-02-26T03:26:26.218 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_readonly_dne PASSED [ 3%] 2026-02-26T03:26:26.859 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_by_id PASSED [ 4%] 2026-02-26T03:26:27.437 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_aio_open PASSED [ 4%] 2026-02-26T03:26:27.500 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_dne PASSED [ 5%] 2026-02-26T03:26:27.508 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_empty PASSED [ 5%] 2026-02-26T03:26:28.051 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list PASSED [ 6%] 2026-02-26T03:26:28.543 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress PASSED [ 6%] 2026-02-26T03:26:29.186 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_canceled PASSED [ 7%] 2026-02-26T03:26:29.673 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress_except PASSED [ 7%] 2026-02-26T03:26:30.496 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename PASSED [ 8%] 2026-02-26T03:26:32.384 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_metadata PASSED [ 8%] 2026-02-26T03:26:32.429 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_config_list PASSED [ 9%] 2026-02-26T03:26:32.466 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_config_set_and_get_and_remove PASSED [ 9%] 2026-02-26T03:26:32.649 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_namespaces PASSED [ 10%] 2026-02-26T03:26:35.952 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_stats PASSED [ 10%] 2026-02-26T03:26:35.953 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_to_string PASSED [ 11%] 2026-02-26T03:26:35.953 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_from_string PASSED [ 11%] 2026-02-26T03:26:36.514 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_update_features SKIPPED [ 12%] 2026-02-26T03:26:37.097 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_with_params SKIPPED [ 12%] 2026-02-26T03:26:37.652 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_id PASSED [ 13%] 2026-02-26T03:26:38.239 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_block_name_prefix PASSED [ 13%] 2026-02-26T03:26:38.813 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_data_pool_id PASSED [ 14%] 2026-02-26T03:26:39.392 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_timestamp PASSED [ 14%] 2026-02-26T03:26:39.957 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_access_timestamp PASSED [ 15%] 2026-02-26T03:26:40.503 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_modify_timestamp PASSED [ 15%] 2026-02-26T03:26:41.196 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_invalidate_cache PASSED [ 16%] 2026-02-26T03:26:41.786 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_stat PASSED [ 16%] 2026-02-26T03:26:42.342 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_flags PASSED [ 17%] 2026-02-26T03:26:42.976 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_auto_close PASSED [ 17%] 2026-02-26T03:26:43.545 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_use_after_close PASSED [ 18%] 2026-02-26T03:26:44.226 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write PASSED [ 18%] 2026-02-26T03:26:44.927 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_with_fadvise_flags PASSED [ 19%] 2026-02-26T03:26:45.604 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes PASSED [ 19%] 2026-02-26T03:26:46.303 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes_thick_provision PASSED [ 20%] 2026-02-26T03:26:46.885 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read PASSED [ 20%] 2026-02-26T03:26:47.458 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_with_fadvise_flags PASSED [ 21%] 2026-02-26T03:26:48.207 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_write PASSED [ 21%] 2026-02-26T03:26:48.796 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_read PASSED [ 22%] 2026-02-26T03:26:49.491 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_read PASSED [ 22%] 2026-02-26T03:26:50.094 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_bad_offset PASSED [ 23%] 2026-02-26T03:26:50.757 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize PASSED [ 23%] 2026-02-26T03:26:51.443 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_allow_shrink_False PASSED [ 24%] 2026-02-26T03:26:55.989 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_size PASSED [ 25%] 2026-02-26T03:26:56.802 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_down PASSED [ 25%] 2026-02-26T03:26:57.658 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_bytes PASSED [ 26%] 2026-02-26T03:26:59.123 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy PASSED [ 26%] 2026-02-26T03:27:00.607 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy2 PASSED [ 27%] 2026-02-26T03:27:01.185 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy3 SKIPPED [ 27%] 2026-02-26T03:27:05.571 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy PASSED [ 28%] 2026-02-26T03:27:13.090 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v1 PASSED [ 28%] 2026-02-26T03:27:20.109 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v2 PASSED [ 29%] 2026-02-26T03:27:27.147 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v1 PASSED [ 29%] 2026-02-26T03:27:34.151 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v2 PASSED [ 30%] 2026-02-26T03:27:41.193 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_flatten PASSED [ 30%] 2026-02-26T03:27:48.009 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_flatten PASSED [ 31%] 2026-02-26T03:27:50.229 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap PASSED [ 31%] 2026-02-26T03:27:52.176 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_exists PASSED [ 32%] 2026-02-26T03:27:57.983 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_flags PASSED [ 32%] 2026-02-26T03:28:02.205 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps PASSED [ 33%] 2026-02-26T03:28:04.213 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps_iterator_auto_close PASSED [ 33%] 2026-02-26T03:28:06.262 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap PASSED [ 34%] 2026-02-26T03:28:06.860 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_not_found PASSED [ 34%] 2026-02-26T03:28:09.282 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap2 PASSED [ 35%] 2026-02-26T03:28:11.300 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_by_id PASSED [ 35%] 2026-02-26T03:28:13.007 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rename_snap PASSED [ 36%] 2026-02-26T03:28:15.285 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_protect_snap PASSED [ 36%] 2026-02-26T03:28:17.288 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_exists PASSED [ 37%] 2026-02-26T03:28:19.969 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_timestamp PASSED [ 37%] 2026-02-26T03:28:27.983 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_limit_snaps PASSED [ 38%] 2026-02-26T03:28:28.725 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_exclusive_lock PASSED [ 38%] 2026-02-26T03:28:29.344 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_snap SKIPPED [ 39%] 2026-02-26T03:28:29.950 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_watcher SKIPPED [ 39%] 2026-02-26T03:28:32.052 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap PASSED [ 40%] 2026-02-26T03:28:34.045 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap_sparse PASSED [ 40%] 2026-02-26T03:28:38.078 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_with_resize PASSED [ 41%] 2026-02-26T03:28:40.089 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap PASSED [ 41%] 2026-02-26T03:28:42.109 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_no_snap PASSED [ 42%] 2026-02-26T03:28:44.108 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_by_id PASSED [ 42%] 2026-02-26T03:28:50.103 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_name PASSED [ 43%] 2026-02-26T03:28:56.132 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_id PASSED [ 43%] 2026-02-26T03:28:58.026 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_sparse PASSED [ 44%] 2026-02-26T03:36:28.528 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_many_snaps PASSED [ 44%] 2026-02-26T03:36:30.607 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_deleted PASSED [ 45%] 2026-02-26T03:36:34.579 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_recreated PASSED [ 45%] 2026-02-26T03:36:35.271 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_lock_unlock PASSED [ 46%] 2026-02-26T03:36:36.820 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_lockers PASSED [ 46%] 2026-02-26T03:36:43.110 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate PASSED [ 47%] 2026-02-26T03:36:46.757 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate_from_trash_snap PASSED [ 47%] 2026-02-26T03:36:49.625 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate_exclude_parent PASSED [ 48%] 2026-02-26T03:36:50.228 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_read PASSED [ 48%] 2026-02-26T03:36:50.976 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write PASSED [ 49%] 2026-02-26T03:36:51.714 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_discard PASSED [ 50%] 2026-02-26T03:36:52.488 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write_zeroes PASSED [ 50%] 2026-02-26T03:36:53.161 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_flush PASSED [ 51%] 2026-02-26T03:36:59.589 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_metadata PASSED [ 51%] 2026-02-26T03:37:00.220 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_watchers_list PASSED [ 52%] 2026-02-26T03:37:01.096 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_config_list PASSED [ 52%] 2026-02-26T03:37:01.972 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_config_set_and_get_and_remove PASSED [ 53%] 2026-02-26T03:37:02.675 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_sparsify PASSED [ 53%] 2026-02-26T03:37:11.973 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks1 PASSED [ 54%] 2026-02-26T03:37:24.965 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks2 PASSED [ 54%] 2026-02-26T03:37:25.657 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_read PASSED [ 55%] 2026-02-26T03:37:26.491 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_write PASSED [ 55%] 2026-02-26T03:37:27.245 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_resize PASSED [ 56%] 2026-02-26T03:37:32.144 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params PASSED [ 56%] 2026-02-26T03:37:37.187 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params2 PASSED [ 57%] 2026-02-26T03:37:40.194 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params3 SKIPPED [ 57%] 2026-02-26T03:37:46.219 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stripe_unit_and_count PASSED [ 58%] 2026-02-26T03:37:50.250 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_format PASSED [ 58%] 2026-02-26T03:37:55.237 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotected PASSED [ 59%] 2026-02-26T03:38:02.240 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotect_with_children PASSED [ 59%] 2026-02-26T03:38:10.275 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_by_snap_id PASSED [ 60%] 2026-02-26T03:38:13.148 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stat PASSED [ 60%] 2026-02-26T03:38:16.295 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_stat PASSED [ 61%] 2026-02-26T03:38:19.258 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_io PASSED [ 61%] 2026-02-26T03:38:22.350 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_read PASSED [ 62%] 2026-02-26T03:38:26.138 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_write PASSED [ 62%] 2026-02-26T03:38:56.147 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_list_children PASSED [ 63%] 2026-02-26T03:39:00.213 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_errors PASSED [ 63%] 2026-02-26T03:39:06.228 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_basic PASSED [ 64%] 2026-02-26T03:39:12.207 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_smaller_order PASSED [ 64%] 2026-02-26T03:39:18.267 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_larger_order PASSED [ 65%] 2026-02-26T03:39:22.264 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_drops_cache PASSED [ 65%] 2026-02-26T03:39:27.246 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_multi_level PASSED [ 66%] 2026-02-26T03:39:31.247 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_with_progress PASSED [ 66%] 2026-02-26T03:39:36.278 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_flatten_multi_level PASSED [ 67%] 2026-02-26T03:39:40.261 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_trash_snapshot PASSED [ 67%] 2026-02-26T03:39:49.271 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks1 PASSED [ 68%] 2026-02-26T03:40:02.308 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks2 PASSED [ 68%] 2026-02-26T03:40:03.200 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_ownership PASSED [ 69%] 2026-02-26T03:40:05.304 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_snapshot_leadership PASSED [ 69%] 2026-02-26T03:40:05.945 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_read_only_leadership PASSED [ 70%] 2026-02-26T03:40:09.313 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_flatten PASSED [ 70%] 2026-02-26T03:40:10.333 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_resize PASSED [ 71%] 2026-02-26T03:40:12.313 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_create PASSED [ 71%] 2026-02-26T03:40:15.314 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_rollback PASSED [ 72%] 2026-02-26T03:40:16.252 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_discard PASSED [ 72%] 2026-02-26T03:40:17.249 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_write PASSED [ 73%] 2026-02-26T03:40:18.020 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_acquire_release_lock PASSED [ 73%] 2026-02-26T03:40:19.346 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_break_lock PASSED [ 74%] 2026-02-26T03:40:20.074 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_uuid PASSED [ 75%] 2026-02-26T03:40:20.841 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_site_name PASSED [ 75%] 2026-02-26T03:40:22.263 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_remote_namespace PASSED [ 76%] 2026-02-26T03:40:23.097 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer_bootstrap PASSED [ 76%] 2026-02-26T03:40:23.880 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer PASSED [ 77%] 2026-02-26T03:40:24.601 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image SKIPPED [ 77%] 2026-02-26T03:40:25.349 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_status SKIPPED [ 78%] 2026-02-26T03:40:34.233 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_create_snapshot PASSED [ 78%] 2026-02-26T03:40:37.946 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_aio_mirror_image_create_snapshot PASSED [ 79%] 2026-02-26T03:40:38.520 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_move PASSED [ 79%] 2026-02-26T03:40:39.608 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_purge PASSED [ 80%] 2026-02-26T03:40:40.139 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_denied PASSED [ 80%] 2026-02-26T03:40:40.675 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove PASSED [ 81%] 2026-02-26T03:40:41.221 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_with_progress PASSED [ 81%] 2026-02-26T03:40:41.756 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_get PASSED [ 82%] 2026-02-26T03:40:42.813 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_list PASSED [ 82%] 2026-02-26T03:40:43.715 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_restore PASSED [ 83%] 2026-02-26T03:40:43.770 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_group PASSED [ 83%] 2026-02-26T03:40:43.838 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename_group PASSED [ 84%] 2026-02-26T03:40:43.841 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_empty PASSED [ 84%] 2026-02-26T03:40:43.895 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups PASSED [ 85%] 2026-02-26T03:40:43.950 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_after_removed PASSED [ 85%] 2026-02-26T03:40:44.741 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_add PASSED [ 86%] 2026-02-26T03:40:45.432 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_empty PASSED [ 86%] 2026-02-26T03:40:46.251 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list PASSED [ 87%] 2026-02-26T03:40:47.214 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_move_to_trash PASSED [ 87%] 2026-02-26T03:40:48.719 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_remove PASSED [ 88%] 2026-02-26T03:40:49.429 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_get_id PASSED [ 88%] 2026-02-26T03:41:03.627 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_many_images PASSED [ 89%] 2026-02-26T03:41:04.561 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_remove PASSED [ 89%] 2026-02-26T03:41:07.931 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info PASSED [ 90%] 2026-02-26T03:41:08.714 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info_no_member_images PASSED [ 90%] 2026-02-26T03:41:09.504 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap PASSED [ 91%] 2026-02-26T03:41:10.421 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_flags PASSED [ 91%] 2026-02-26T03:41:52.044 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_list_many PASSED [ 92%] 2026-02-26T03:41:54.389 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_namespace PASSED [ 92%] 2026-02-26T03:41:55.192 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rename PASSED [ 93%] 2026-02-26T03:41:58.654 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone PASSED [ 93%] 2026-02-26T03:42:02.211 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone_flatten PASSED [ 94%] 2026-02-26T03:42:05.331 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_diff_iterate PASSED [ 94%] 2026-02-26T03:42:18.587 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rollback PASSED [ 95%] 2026-02-26T03:42:20.852 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration PASSED [ 95%] 2026-02-26T03:42:26.195 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v1 PASSED [ 96%] 2026-02-26T03:42:31.246 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v2 PASSED [ 96%] 2026-02-26T03:42:36.229 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v1 PASSED [ 97%] 2026-02-26T03:42:41.246 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v2 PASSED [ 97%] 2026-02-26T03:42:46.798 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_import PASSED [ 98%] 2026-02-26T03:42:48.804 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_with_progress PASSED [ 98%] 2026-02-26T03:42:50.451 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort PASSED [ 99%] 2026-02-26T03:42:52.660 INFO:tasks.workunit.client.0.trial095.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort_with_progress PASSED [100%] 2026-02-26T03:42:52.660 INFO:tasks.workunit.client.0.trial095.stdout: 2026-02-26T03:42:52.660 INFO:tasks.workunit.client.0.trial095.stdout:=============================== warnings summary =============================== 2026-02-26T03:42:52.660 INFO:tasks.workunit.client.0.trial095.stdout:test_rbd.py::test_remove_with_progress_except 2026-02-26T03:42:52.660 INFO:tasks.workunit.client.0.trial095.stdout: /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: 'rbd.progress_callback' 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: Traceback (most recent call last): 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rbd.py", line 430, in progress_cb 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: raise Exception() 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: Exception 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout: 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout:-- Docs: https://docs.pytest.org/en/stable/warnings.html 2026-02-26T03:42:52.661 INFO:tasks.workunit.client.0.trial095.stdout:============ 188 passed, 8 skipped, 1 warning in 1000.20s (0:16:40) ============ 2026-02-26T03:42:52.695 INFO:tasks.workunit.client.0.trial095.stderr:+ exit 0 2026-02-26T03:42:52.696 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-26T03:42:52.697 DEBUG:teuthology.orchestra.run.trial095:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-26T03:42:52.765 INFO:tasks.workunit:Stopping ['rbd/test_librbd_python.sh'] on client.0... 2026-02-26T03:42:52.766 DEBUG:teuthology.orchestra.run.trial095:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-26T03:42:53.178 DEBUG:teuthology.parallel:result is None 2026-02-26T03:42:53.178 DEBUG:teuthology.orchestra.run.trial095:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:42:53.204 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:42:53.204 DEBUG:teuthology.orchestra.run.trial095:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:42:53.261 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:42:53.261 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:42:53.278 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:42:53.278 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:42:53.376 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.372+0000 7f95c8f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:53.378 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.377+0000 7f95c8f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:53.491 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:42:53.491 INFO:teuthology.orchestra.run.trial095.stderr:dumped all 2026-02-26T03:42:53.500 INFO:teuthology.orchestra.run.trial095.stdout:{"pg_ready":true,"pg_map":{"version":1254,"stamp":"2026-02-26T03:42:52.653862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62939104,"kb_used_data":62939104,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94347248,"statfs":{"total":161061224448,"available":96611581952,"internally_reserved":0,"allocated":64449642496,"data_stored":64449642496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-390369,"num_read_kb":-50025,"num_write":-13185,"num_write_kb":-80785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.011225"},"pg_stats":[{"pgid":"1.0","version":"11'52","reported_seq":2271,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.564445+0000","last_change":"2026-02-26T03:25:39.742660+0000","last_active":"2026-02-26T03:42:47.564445+0000","last_peered":"2026-02-26T03:42:47.564445+0000","last_clean":"2026-02-26T03:42:47.564445+0000","last_became_active":"2026-02-26T03:25:39.733389+0000","last_became_peered":"2026-02-26T03:25:39.733389+0000","last_unstale":"2026-02-26T03:42:47.564445+0000","last_undegraded":"2026-02-26T03:42:47.564445+0000","last_fullsized":"2026-02-26T03:42:47.564445+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_clean_scrub_stamp":"2026-02-26T03:25:38.697555+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":2197,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567971+0000","last_change":"2026-02-26T03:25:46.741695+0000","last_active":"2026-02-26T03:42:47.567971+0000","last_peered":"2026-02-26T03:42:47.567971+0000","last_clean":"2026-02-26T03:42:47.567971+0000","last_became_active":"2026-02-26T03:25:44.758142+0000","last_became_peered":"2026-02-26T03:25:44.758142+0000","last_unstale":"2026-02-26T03:42:47.567971+0000","last_undegraded":"2026-02-26T03:42:47.567971+0000","last_fullsized":"2026-02-26T03:42:47.567971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.563185+0000","last_change":"2026-02-26T03:25:46.737785+0000","last_active":"2026-02-26T03:42:47.563185+0000","last_peered":"2026-02-26T03:42:47.563185+0000","last_clean":"2026-02-26T03:42:47.563185+0000","last_became_active":"2026-02-26T03:25:44.775206+0000","last_became_peered":"2026-02-26T03:25:44.775206+0000","last_unstale":"2026-02-26T03:42:47.563185+0000","last_undegraded":"2026-02-26T03:42:47.563185+0000","last_fullsized":"2026-02-26T03:42:47.563185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566394+0000","last_change":"2026-02-26T03:25:46.739851+0000","last_active":"2026-02-26T03:42:47.566394+0000","last_peered":"2026-02-26T03:42:47.566394+0000","last_clean":"2026-02-26T03:42:47.566394+0000","last_became_active":"2026-02-26T03:25:44.780653+0000","last_became_peered":"2026-02-26T03:25:44.780653+0000","last_unstale":"2026-02-26T03:42:47.566394+0000","last_undegraded":"2026-02-26T03:42:47.566394+0000","last_fullsized":"2026-02-26T03:42:47.566394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":2198,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566821+0000","last_change":"2026-02-26T03:25:46.741963+0000","last_active":"2026-02-26T03:42:47.566821+0000","last_peered":"2026-02-26T03:42:47.566821+0000","last_clean":"2026-02-26T03:42:47.566821+0000","last_became_active":"2026-02-26T03:25:44.770887+0000","last_became_peered":"2026-02-26T03:25:44.770887+0000","last_unstale":"2026-02-26T03:42:47.566821+0000","last_undegraded":"2026-02-26T03:42:47.566821+0000","last_fullsized":"2026-02-26T03:42:47.566821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567343+0000","last_change":"2026-02-26T03:25:46.739713+0000","last_active":"2026-02-26T03:42:47.567343+0000","last_peered":"2026-02-26T03:42:47.567343+0000","last_clean":"2026-02-26T03:42:47.567343+0000","last_became_active":"2026-02-26T03:25:44.764783+0000","last_became_peered":"2026-02-26T03:25:44.764783+0000","last_unstale":"2026-02-26T03:42:47.567343+0000","last_undegraded":"2026-02-26T03:42:47.567343+0000","last_fullsized":"2026-02-26T03:42:47.567343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567591+0000","last_change":"2026-02-26T03:25:46.741334+0000","last_active":"2026-02-26T03:42:47.567591+0000","last_peered":"2026-02-26T03:42:47.567591+0000","last_clean":"2026-02-26T03:42:47.567591+0000","last_became_active":"2026-02-26T03:25:44.769736+0000","last_became_peered":"2026-02-26T03:25:44.769736+0000","last_unstale":"2026-02-26T03:42:47.567591+0000","last_undegraded":"2026-02-26T03:42:47.567591+0000","last_fullsized":"2026-02-26T03:42:47.567591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.568194+0000","last_change":"2026-02-26T03:25:46.740678+0000","last_active":"2026-02-26T03:42:47.568194+0000","last_peered":"2026-02-26T03:42:47.568194+0000","last_clean":"2026-02-26T03:42:47.568194+0000","last_became_active":"2026-02-26T03:25:44.758719+0000","last_became_peered":"2026-02-26T03:25:44.758719+0000","last_unstale":"2026-02-26T03:42:47.568194+0000","last_undegraded":"2026-02-26T03:42:47.568194+0000","last_fullsized":"2026-02-26T03:42:47.568194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566452+0000","last_change":"2026-02-26T03:25:46.738834+0000","last_active":"2026-02-26T03:42:47.566452+0000","last_peered":"2026-02-26T03:42:47.566452+0000","last_clean":"2026-02-26T03:42:47.566452+0000","last_became_active":"2026-02-26T03:25:44.759336+0000","last_became_peered":"2026-02-26T03:25:44.759336+0000","last_unstale":"2026-02-26T03:42:47.566452+0000","last_undegraded":"2026-02-26T03:42:47.566452+0000","last_fullsized":"2026-02-26T03:42:47.566452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96611520512,"internally_reserved":0,"allocated":64449703936,"data_stored":64449703936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20978108,"kb_used_data":20978108,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31450676,"statfs":{"total":53687074816,"available":32205492224,"internally_reserved":0,"allocated":21481582592,"data_stored":21481582592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20981256,"kb_used_data":20981256,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31447528,"statfs":{"total":53687074816,"available":32202268672,"internally_reserved":0,"allocated":21484806144,"data_stored":21484806144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20979740,"kb_used_data":20979740,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31449044,"statfs":{"total":53687074816,"available":32203821056,"internally_reserved":0,"allocated":21483253760,"data_stored":21483253760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32203743232,"internally_reserved":0,"allocated":21483331584,"data_stored":21483331584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32205664256,"internally_reserved":0,"allocated":21481410560,"data_stored":21481410560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:42:53.502 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:42:53.578 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.575+0000 7fb67285a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:53.580 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.580+0000 7fb67285a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:53.692 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:42:53.693 INFO:teuthology.orchestra.run.trial095.stderr:dumped all 2026-02-26T03:42:53.702 INFO:teuthology.orchestra.run.trial095.stdout:{"pg_ready":true,"pg_map":{"version":1254,"stamp":"2026-02-26T03:42:52.653862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62939104,"kb_used_data":62939104,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94347248,"statfs":{"total":161061224448,"available":96611581952,"internally_reserved":0,"allocated":64449642496,"data_stored":64449642496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-390369,"num_read_kb":-50025,"num_write":-13185,"num_write_kb":-80785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.011225"},"pg_stats":[{"pgid":"1.0","version":"11'52","reported_seq":2271,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.564445+0000","last_change":"2026-02-26T03:25:39.742660+0000","last_active":"2026-02-26T03:42:47.564445+0000","last_peered":"2026-02-26T03:42:47.564445+0000","last_clean":"2026-02-26T03:42:47.564445+0000","last_became_active":"2026-02-26T03:25:39.733389+0000","last_became_peered":"2026-02-26T03:25:39.733389+0000","last_unstale":"2026-02-26T03:42:47.564445+0000","last_undegraded":"2026-02-26T03:42:47.564445+0000","last_fullsized":"2026-02-26T03:42:47.564445+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_clean_scrub_stamp":"2026-02-26T03:25:38.697555+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":2197,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567971+0000","last_change":"2026-02-26T03:25:46.741695+0000","last_active":"2026-02-26T03:42:47.567971+0000","last_peered":"2026-02-26T03:42:47.567971+0000","last_clean":"2026-02-26T03:42:47.567971+0000","last_became_active":"2026-02-26T03:25:44.758142+0000","last_became_peered":"2026-02-26T03:25:44.758142+0000","last_unstale":"2026-02-26T03:42:47.567971+0000","last_undegraded":"2026-02-26T03:42:47.567971+0000","last_fullsized":"2026-02-26T03:42:47.567971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.563185+0000","last_change":"2026-02-26T03:25:46.737785+0000","last_active":"2026-02-26T03:42:47.563185+0000","last_peered":"2026-02-26T03:42:47.563185+0000","last_clean":"2026-02-26T03:42:47.563185+0000","last_became_active":"2026-02-26T03:25:44.775206+0000","last_became_peered":"2026-02-26T03:25:44.775206+0000","last_unstale":"2026-02-26T03:42:47.563185+0000","last_undegraded":"2026-02-26T03:42:47.563185+0000","last_fullsized":"2026-02-26T03:42:47.563185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566394+0000","last_change":"2026-02-26T03:25:46.739851+0000","last_active":"2026-02-26T03:42:47.566394+0000","last_peered":"2026-02-26T03:42:47.566394+0000","last_clean":"2026-02-26T03:42:47.566394+0000","last_became_active":"2026-02-26T03:25:44.780653+0000","last_became_peered":"2026-02-26T03:25:44.780653+0000","last_unstale":"2026-02-26T03:42:47.566394+0000","last_undegraded":"2026-02-26T03:42:47.566394+0000","last_fullsized":"2026-02-26T03:42:47.566394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":2198,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566821+0000","last_change":"2026-02-26T03:25:46.741963+0000","last_active":"2026-02-26T03:42:47.566821+0000","last_peered":"2026-02-26T03:42:47.566821+0000","last_clean":"2026-02-26T03:42:47.566821+0000","last_became_active":"2026-02-26T03:25:44.770887+0000","last_became_peered":"2026-02-26T03:25:44.770887+0000","last_unstale":"2026-02-26T03:42:47.566821+0000","last_undegraded":"2026-02-26T03:42:47.566821+0000","last_fullsized":"2026-02-26T03:42:47.566821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567343+0000","last_change":"2026-02-26T03:25:46.739713+0000","last_active":"2026-02-26T03:42:47.567343+0000","last_peered":"2026-02-26T03:42:47.567343+0000","last_clean":"2026-02-26T03:42:47.567343+0000","last_became_active":"2026-02-26T03:25:44.764783+0000","last_became_peered":"2026-02-26T03:25:44.764783+0000","last_unstale":"2026-02-26T03:42:47.567343+0000","last_undegraded":"2026-02-26T03:42:47.567343+0000","last_fullsized":"2026-02-26T03:42:47.567343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567591+0000","last_change":"2026-02-26T03:25:46.741334+0000","last_active":"2026-02-26T03:42:47.567591+0000","last_peered":"2026-02-26T03:42:47.567591+0000","last_clean":"2026-02-26T03:42:47.567591+0000","last_became_active":"2026-02-26T03:25:44.769736+0000","last_became_peered":"2026-02-26T03:25:44.769736+0000","last_unstale":"2026-02-26T03:42:47.567591+0000","last_undegraded":"2026-02-26T03:42:47.567591+0000","last_fullsized":"2026-02-26T03:42:47.567591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.568194+0000","last_change":"2026-02-26T03:25:46.740678+0000","last_active":"2026-02-26T03:42:47.568194+0000","last_peered":"2026-02-26T03:42:47.568194+0000","last_clean":"2026-02-26T03:42:47.568194+0000","last_became_active":"2026-02-26T03:25:44.758719+0000","last_became_peered":"2026-02-26T03:25:44.758719+0000","last_unstale":"2026-02-26T03:42:47.568194+0000","last_undegraded":"2026-02-26T03:42:47.568194+0000","last_fullsized":"2026-02-26T03:42:47.568194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566452+0000","last_change":"2026-02-26T03:25:46.738834+0000","last_active":"2026-02-26T03:42:47.566452+0000","last_peered":"2026-02-26T03:42:47.566452+0000","last_clean":"2026-02-26T03:42:47.566452+0000","last_became_active":"2026-02-26T03:25:44.759336+0000","last_became_peered":"2026-02-26T03:25:44.759336+0000","last_unstale":"2026-02-26T03:42:47.566452+0000","last_undegraded":"2026-02-26T03:42:47.566452+0000","last_fullsized":"2026-02-26T03:42:47.566452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96611520512,"internally_reserved":0,"allocated":64449703936,"data_stored":64449703936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20978108,"kb_used_data":20978108,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31450676,"statfs":{"total":53687074816,"available":32205492224,"internally_reserved":0,"allocated":21481582592,"data_stored":21481582592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20981256,"kb_used_data":20981256,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31447528,"statfs":{"total":53687074816,"available":32202268672,"internally_reserved":0,"allocated":21484806144,"data_stored":21484806144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20979740,"kb_used_data":20979740,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31449044,"statfs":{"total":53687074816,"available":32203821056,"internally_reserved":0,"allocated":21483253760,"data_stored":21483253760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32203743232,"internally_reserved":0,"allocated":21483331584,"data_stored":21483331584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32205664256,"internally_reserved":0,"allocated":21481410560,"data_stored":21481410560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:42:53.704 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:42:53.704 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:42:53.779 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.775+0000 7f78eab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:53.781 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.780+0000 7f78eab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:53.891 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:42:53.892 INFO:teuthology.orchestra.run.trial095.stderr:dumped all 2026-02-26T03:42:53.902 INFO:teuthology.orchestra.run.trial095.stdout:{"pg_ready":true,"pg_map":{"version":1254,"stamp":"2026-02-26T03:42:52.653862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62939104,"kb_used_data":62939104,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94347248,"statfs":{"total":161061224448,"available":96611581952,"internally_reserved":0,"allocated":64449642496,"data_stored":64449642496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-390369,"num_read_kb":-50025,"num_write":-13185,"num_write_kb":-80785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.011225"},"pg_stats":[{"pgid":"1.0","version":"11'52","reported_seq":2271,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.564445+0000","last_change":"2026-02-26T03:25:39.742660+0000","last_active":"2026-02-26T03:42:47.564445+0000","last_peered":"2026-02-26T03:42:47.564445+0000","last_clean":"2026-02-26T03:42:47.564445+0000","last_became_active":"2026-02-26T03:25:39.733389+0000","last_became_peered":"2026-02-26T03:25:39.733389+0000","last_unstale":"2026-02-26T03:42:47.564445+0000","last_undegraded":"2026-02-26T03:42:47.564445+0000","last_fullsized":"2026-02-26T03:42:47.564445+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_clean_scrub_stamp":"2026-02-26T03:25:38.697555+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":2197,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567971+0000","last_change":"2026-02-26T03:25:46.741695+0000","last_active":"2026-02-26T03:42:47.567971+0000","last_peered":"2026-02-26T03:42:47.567971+0000","last_clean":"2026-02-26T03:42:47.567971+0000","last_became_active":"2026-02-26T03:25:44.758142+0000","last_became_peered":"2026-02-26T03:25:44.758142+0000","last_unstale":"2026-02-26T03:42:47.567971+0000","last_undegraded":"2026-02-26T03:42:47.567971+0000","last_fullsized":"2026-02-26T03:42:47.567971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.563185+0000","last_change":"2026-02-26T03:25:46.737785+0000","last_active":"2026-02-26T03:42:47.563185+0000","last_peered":"2026-02-26T03:42:47.563185+0000","last_clean":"2026-02-26T03:42:47.563185+0000","last_became_active":"2026-02-26T03:25:44.775206+0000","last_became_peered":"2026-02-26T03:25:44.775206+0000","last_unstale":"2026-02-26T03:42:47.563185+0000","last_undegraded":"2026-02-26T03:42:47.563185+0000","last_fullsized":"2026-02-26T03:42:47.563185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566394+0000","last_change":"2026-02-26T03:25:46.739851+0000","last_active":"2026-02-26T03:42:47.566394+0000","last_peered":"2026-02-26T03:42:47.566394+0000","last_clean":"2026-02-26T03:42:47.566394+0000","last_became_active":"2026-02-26T03:25:44.780653+0000","last_became_peered":"2026-02-26T03:25:44.780653+0000","last_unstale":"2026-02-26T03:42:47.566394+0000","last_undegraded":"2026-02-26T03:42:47.566394+0000","last_fullsized":"2026-02-26T03:42:47.566394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":2198,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566821+0000","last_change":"2026-02-26T03:25:46.741963+0000","last_active":"2026-02-26T03:42:47.566821+0000","last_peered":"2026-02-26T03:42:47.566821+0000","last_clean":"2026-02-26T03:42:47.566821+0000","last_became_active":"2026-02-26T03:25:44.770887+0000","last_became_peered":"2026-02-26T03:25:44.770887+0000","last_unstale":"2026-02-26T03:42:47.566821+0000","last_undegraded":"2026-02-26T03:42:47.566821+0000","last_fullsized":"2026-02-26T03:42:47.566821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567343+0000","last_change":"2026-02-26T03:25:46.739713+0000","last_active":"2026-02-26T03:42:47.567343+0000","last_peered":"2026-02-26T03:42:47.567343+0000","last_clean":"2026-02-26T03:42:47.567343+0000","last_became_active":"2026-02-26T03:25:44.764783+0000","last_became_peered":"2026-02-26T03:25:44.764783+0000","last_unstale":"2026-02-26T03:42:47.567343+0000","last_undegraded":"2026-02-26T03:42:47.567343+0000","last_fullsized":"2026-02-26T03:42:47.567343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.567591+0000","last_change":"2026-02-26T03:25:46.741334+0000","last_active":"2026-02-26T03:42:47.567591+0000","last_peered":"2026-02-26T03:42:47.567591+0000","last_clean":"2026-02-26T03:42:47.567591+0000","last_became_active":"2026-02-26T03:25:44.769736+0000","last_became_peered":"2026-02-26T03:25:44.769736+0000","last_unstale":"2026-02-26T03:42:47.567591+0000","last_undegraded":"2026-02-26T03:42:47.567591+0000","last_fullsized":"2026-02-26T03:42:47.567591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.568194+0000","last_change":"2026-02-26T03:25:46.740678+0000","last_active":"2026-02-26T03:42:47.568194+0000","last_peered":"2026-02-26T03:42:47.568194+0000","last_clean":"2026-02-26T03:42:47.568194+0000","last_became_active":"2026-02-26T03:25:44.758719+0000","last_became_peered":"2026-02-26T03:25:44.758719+0000","last_unstale":"2026-02-26T03:42:47.568194+0000","last_undegraded":"2026-02-26T03:42:47.568194+0000","last_fullsized":"2026-02-26T03:42:47.568194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2196,"reported_epoch":743,"state":"active+clean","last_fresh":"2026-02-26T03:42:47.566452+0000","last_change":"2026-02-26T03:25:46.738834+0000","last_active":"2026-02-26T03:42:47.566452+0000","last_peered":"2026-02-26T03:42:47.566452+0000","last_clean":"2026-02-26T03:42:47.566452+0000","last_became_active":"2026-02-26T03:25:44.759336+0000","last_became_peered":"2026-02-26T03:25:44.759336+0000","last_unstale":"2026-02-26T03:42:47.566452+0000","last_undegraded":"2026-02-26T03:42:47.566452+0000","last_fullsized":"2026-02-26T03:42:47.566452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96611520512,"internally_reserved":0,"allocated":64449703936,"data_stored":64449703936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20978108,"kb_used_data":20978108,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31450676,"statfs":{"total":53687074816,"available":32205492224,"internally_reserved":0,"allocated":21481582592,"data_stored":21481582592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20981256,"kb_used_data":20981256,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31447528,"statfs":{"total":53687074816,"available":32202268672,"internally_reserved":0,"allocated":21484806144,"data_stored":21484806144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20979740,"kb_used_data":20979740,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31449044,"statfs":{"total":53687074816,"available":32203821056,"internally_reserved":0,"allocated":21483253760,"data_stored":21483253760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32203743232,"internally_reserved":0,"allocated":21483331584,"data_stored":21483331584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32202113024,"internally_reserved":0,"allocated":21484961792,"data_stored":21484961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32205664256,"internally_reserved":0,"allocated":21481410560,"data_stored":21481410560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:42:53.903 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:42:53.980 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.976+0000 7f4d3885a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:53.983 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:53.982+0000 7f4d375d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:54.099 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:42:54.099 INFO:teuthology.orchestra.run.trial095.stdout:{"epoch":744,"fsid":"efb8f16e-da4e-4b0e-bd52-2a50c31c1d6e","created":"2026-02-26T03:25:27.396677+0000","modified":"2026-02-26T03:42:52.652974+0000","last_up_change":"2026-02-26T03:25:38.697555+0000","last_in_change":"2026-02-26T03:25:29.154553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:37.853178+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:25:43.083700+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c24bac03-1167-4daf-8c9b-92e20facb087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":590,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6804","nonce":3851961937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6805","nonce":3851961937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6806","nonce":3851961937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6809","nonce":3851961937}]},"public_addr":"10.20.193.95:6804/3851961937","cluster_addr":"10.20.193.95:6805/3851961937","heartbeat_back_addr":"10.20.193.95:6806/3851961937","heartbeat_front_addr":"10.20.193.95:6809/3851961937","state":["exists","up"]},{"osd":1,"uuid":"0033ac36-5e4b-47ed-b8d7-fe23edfa432d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":590,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6802","nonce":1661387041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6803","nonce":1661387041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6810","nonce":1661387041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6811","nonce":1661387041}]},"public_addr":"10.20.193.95:6802/1661387041","cluster_addr":"10.20.193.95:6803/1661387041","heartbeat_back_addr":"10.20.193.95:6810/1661387041","heartbeat_front_addr":"10.20.193.95:6811/1661387041","state":["exists","up"]},{"osd":2,"uuid":"a085a950-c9a9-46a1-a73d-897375a15a1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":590,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6808","nonce":2801899040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6807","nonce":2801899040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6814","nonce":2801899040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.95:6813","nonce":2801899040}]},"public_addr":"10.20.193.95:6808/2801899040","cluster_addr":"10.20.193.95:6807/2801899040","heartbeat_back_addr":"10.20.193.95:6814/2801899040","heartbeat_front_addr":"10.20.193.95:6813/2801899040","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.95:0/3323478674":"2026-02-26T04:40:18.302879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":369}]},{"pool":4,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:42:55.110 INFO:tasks.ceph:Scrubbing osd.0 2026-02-26T03:42:55.111 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:42:55.191 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.185+0000 7f2830e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.193 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.192+0000 7f2830e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.202 INFO:teuthology.orchestra.run.trial095.stdout:{ 2026-02-26T03:42:55.202 INFO:teuthology.orchestra.run.trial095.stdout: "success": "" 2026-02-26T03:42:55.202 INFO:teuthology.orchestra.run.trial095.stdout:} 2026-02-26T03:42:55.210 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-26T03:42:55.287 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.283+0000 7fcba330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.290 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.289+0000 7fcba330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.402 INFO:teuthology.orchestra.run.trial095.stderr:instructed osd(s) 0 to deep-scrub 2026-02-26T03:42:55.412 INFO:tasks.ceph:Scrubbing osd.1 2026-02-26T03:42:55.412 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:42:55.491 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.485+0000 7fb4e4d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.493 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.492+0000 7fb4e4d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.503 INFO:teuthology.orchestra.run.trial095.stdout:{ 2026-02-26T03:42:55.503 INFO:teuthology.orchestra.run.trial095.stdout: "success": "" 2026-02-26T03:42:55.503 INFO:teuthology.orchestra.run.trial095.stdout:} 2026-02-26T03:42:55.510 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-26T03:42:55.586 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.582+0000 7f9ca810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.588 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.587+0000 7f9ca810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.700 INFO:teuthology.orchestra.run.trial095.stderr:instructed osd(s) 1 to deep-scrub 2026-02-26T03:42:55.712 INFO:tasks.ceph:Scrubbing osd.2 2026-02-26T03:42:55.712 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:42:55.792 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.786+0000 7f775e011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.794 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.793+0000 7f775e011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.803 INFO:teuthology.orchestra.run.trial095.stdout:{ 2026-02-26T03:42:55.803 INFO:teuthology.orchestra.run.trial095.stdout: "success": "" 2026-02-26T03:42:55.803 INFO:teuthology.orchestra.run.trial095.stdout:} 2026-02-26T03:42:55.810 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-26T03:42:55.888 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.884+0000 7fcf543a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:55.890 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:55.889+0000 7fcf543a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:56.001 INFO:teuthology.orchestra.run.trial095.stderr:instructed osd(s) 2 to deep-scrub 2026-02-26T03:42:56.011 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:42:56.088 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:56.085+0000 7fb216bcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:56.091 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:42:56.090+0000 7fb216bcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:56.201 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:42:56.202 INFO:teuthology.orchestra.run.trial095.stderr:dumped all 2026-02-26T03:42:56.210 INFO:teuthology.orchestra.run.trial095.stdout:{"pg_ready":true,"pg_map":{"version":1256,"stamp":"2026-02-26T03:42:56.003236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62933572,"kb_used_data":62933572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94352780,"statfs":{"total":161061224448,"available":96617246720,"internally_reserved":0,"allocated":64443977728,"data_stored":64443977728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-390369,"num_read_kb":-50025,"num_write":-13185,"num_write_kb":-80785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.450925"},"pg_stats":[{"pgid":"1.0","version":"11'52","reported_seq":2274,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.683206+0000","last_change":"2026-02-26T03:25:39.742660+0000","last_active":"2026-02-26T03:42:52.683206+0000","last_peered":"2026-02-26T03:42:52.683206+0000","last_clean":"2026-02-26T03:42:52.683206+0000","last_became_active":"2026-02-26T03:25:39.733389+0000","last_became_peered":"2026-02-26T03:25:39.733389+0000","last_unstale":"2026-02-26T03:42:52.683206+0000","last_undegraded":"2026-02-26T03:42:52.683206+0000","last_fullsized":"2026-02-26T03:42:52.683206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:38.697555+0000","last_clean_scrub_stamp":"2026-02-26T03:25:38.697555+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":2200,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.682721+0000","last_change":"2026-02-26T03:25:46.741695+0000","last_active":"2026-02-26T03:42:52.682721+0000","last_peered":"2026-02-26T03:42:52.682721+0000","last_clean":"2026-02-26T03:42:52.682721+0000","last_became_active":"2026-02-26T03:25:44.758142+0000","last_became_peered":"2026-02-26T03:25:44.758142+0000","last_unstale":"2026-02-26T03:42:52.682721+0000","last_undegraded":"2026-02-26T03:42:52.682721+0000","last_fullsized":"2026-02-26T03:42:52.682721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2199,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.671956+0000","last_change":"2026-02-26T03:25:46.737785+0000","last_active":"2026-02-26T03:42:52.671956+0000","last_peered":"2026-02-26T03:42:52.671956+0000","last_clean":"2026-02-26T03:42:52.671956+0000","last_became_active":"2026-02-26T03:25:44.775206+0000","last_became_peered":"2026-02-26T03:25:44.775206+0000","last_unstale":"2026-02-26T03:42:52.671956+0000","last_undegraded":"2026-02-26T03:42:52.671956+0000","last_fullsized":"2026-02-26T03:42:52.671956+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2199,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.683474+0000","last_change":"2026-02-26T03:25:46.739851+0000","last_active":"2026-02-26T03:42:52.683474+0000","last_peered":"2026-02-26T03:42:52.683474+0000","last_clean":"2026-02-26T03:42:52.683474+0000","last_became_active":"2026-02-26T03:25:44.780653+0000","last_became_peered":"2026-02-26T03:25:44.780653+0000","last_unstale":"2026-02-26T03:42:52.683474+0000","last_undegraded":"2026-02-26T03:42:52.683474+0000","last_fullsized":"2026-02-26T03:42:52.683474+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":2201,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.684784+0000","last_change":"2026-02-26T03:25:46.741963+0000","last_active":"2026-02-26T03:42:52.684784+0000","last_peered":"2026-02-26T03:42:52.684784+0000","last_clean":"2026-02-26T03:42:52.684784+0000","last_became_active":"2026-02-26T03:25:44.770887+0000","last_became_peered":"2026-02-26T03:25:44.770887+0000","last_unstale":"2026-02-26T03:42:52.684784+0000","last_undegraded":"2026-02-26T03:42:52.684784+0000","last_fullsized":"2026-02-26T03:42:52.684784+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2199,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.683120+0000","last_change":"2026-02-26T03:25:46.739713+0000","last_active":"2026-02-26T03:42:52.683120+0000","last_peered":"2026-02-26T03:42:52.683120+0000","last_clean":"2026-02-26T03:42:52.683120+0000","last_became_active":"2026-02-26T03:25:44.764783+0000","last_became_peered":"2026-02-26T03:25:44.764783+0000","last_unstale":"2026-02-26T03:42:52.683120+0000","last_undegraded":"2026-02-26T03:42:52.683120+0000","last_fullsized":"2026-02-26T03:42:52.683120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2199,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.692042+0000","last_change":"2026-02-26T03:25:46.741334+0000","last_active":"2026-02-26T03:42:52.692042+0000","last_peered":"2026-02-26T03:42:52.692042+0000","last_clean":"2026-02-26T03:42:52.692042+0000","last_became_active":"2026-02-26T03:25:44.769736+0000","last_became_peered":"2026-02-26T03:25:44.769736+0000","last_unstale":"2026-02-26T03:42:52.692042+0000","last_undegraded":"2026-02-26T03:42:52.692042+0000","last_fullsized":"2026-02-26T03:42:52.692042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2199,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.684313+0000","last_change":"2026-02-26T03:25:46.740678+0000","last_active":"2026-02-26T03:42:52.684313+0000","last_peered":"2026-02-26T03:42:52.684313+0000","last_clean":"2026-02-26T03:42:52.684313+0000","last_became_active":"2026-02-26T03:25:44.758719+0000","last_became_peered":"2026-02-26T03:25:44.758719+0000","last_unstale":"2026-02-26T03:42:52.684313+0000","last_undegraded":"2026-02-26T03:42:52.684313+0000","last_fullsized":"2026-02-26T03:42:52.684313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2199,"reported_epoch":744,"state":"active+clean","last_fresh":"2026-02-26T03:42:52.680986+0000","last_change":"2026-02-26T03:25:46.738834+0000","last_active":"2026-02-26T03:42:52.680986+0000","last_peered":"2026-02-26T03:42:52.680986+0000","last_clean":"2026-02-26T03:42:52.680986+0000","last_became_active":"2026-02-26T03:25:44.759336+0000","last_became_peered":"2026-02-26T03:25:44.759336+0000","last_unstale":"2026-02-26T03:42:52.680986+0000","last_undegraded":"2026-02-26T03:42:52.680986+0000","last_fullsized":"2026-02-26T03:42:52.680986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:43.714753+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96617340928,"internally_reserved":0,"allocated":64443883520,"data_stored":64443883520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32202317824,"internally_reserved":0,"allocated":21484756992,"data_stored":21484756992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972532,"kb_used_data":20972532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456252,"statfs":{"total":53687074816,"available":32211202048,"internally_reserved":0,"allocated":21475872768,"data_stored":21475872768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20981228,"kb_used_data":20981228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31447556,"statfs":{"total":53687074816,"available":32202297344,"internally_reserved":0,"allocated":21484777472,"data_stored":21484777472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20979812,"kb_used_data":20979812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31448972,"statfs":{"total":53687074816,"available":32203747328,"internally_reserved":0,"allocated":21483327488,"data_stored":21483327488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32202317824,"internally_reserved":0,"allocated":21484756992,"data_stored":21484756992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32203821056,"internally_reserved":0,"allocated":21483253760,"data_stored":21483253760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32202317824,"internally_reserved":0,"allocated":21484756992,"data_stored":21484756992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211202048,"internally_reserved":0,"allocated":21475872768,"data_stored":21475872768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:42:56.216 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-26T03:25:38.697555+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=38, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.216 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.217 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.217 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.217 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.217 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.217 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.217 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.218 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-26T03:25:43.714753+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=43, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=42, tm_sec=54, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:42:56.218 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-26T03:43:16.219 DEBUG:teuthology.orchestra.run.trial095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:43:16.299 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:43:16.295+0000 7fe78e1cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:16.302 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:43:16.300+0000 7fe78e1cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:16.411 INFO:teuthology.orchestra.run.trial095.stdout: 2026-02-26T03:43:16.412 INFO:teuthology.orchestra.run.trial095.stderr:dumped all 2026-02-26T03:43:16.422 INFO:teuthology.orchestra.run.trial095.stdout:{"pg_ready":true,"pg_map":{"version":1267,"stamp":"2026-02-26T03:43:16.006604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62919172,"kb_used_data":62919172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94367180,"statfs":{"total":161061224448,"available":96631992320,"internally_reserved":0,"allocated":64429232128,"data_stored":64429232128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002002"},"pg_stats":[{"pgid":"1.0","version":"11'52","reported_seq":2279,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.566250+0000","last_change":"2026-02-26T03:42:55.726565+0000","last_active":"2026-02-26T03:42:57.566250+0000","last_peered":"2026-02-26T03:42:57.566250+0000","last_clean":"2026-02-26T03:42:57.566250+0000","last_became_active":"2026-02-26T03:25:39.733389+0000","last_became_peered":"2026-02-26T03:25:39.733389+0000","last_unstale":"2026-02-26T03:42:57.566250+0000","last_undegraded":"2026-02-26T03:42:57.566250+0000","last_fullsized":"2026-02-26T03:42:57.566250+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'52","last_scrub_stamp":"2026-02-26T03:42:55.726496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:55.726496+0000","last_clean_scrub_stamp":"2026-02-26T03:25:38.697555+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":2205,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.567591+0000","last_change":"2026-02-26T03:42:55.715558+0000","last_active":"2026-02-26T03:42:57.567591+0000","last_peered":"2026-02-26T03:42:57.567591+0000","last_clean":"2026-02-26T03:42:57.567591+0000","last_became_active":"2026-02-26T03:25:44.758142+0000","last_became_peered":"2026-02-26T03:25:44.758142+0000","last_unstale":"2026-02-26T03:42:57.567591+0000","last_undegraded":"2026-02-26T03:42:57.567591+0000","last_fullsized":"2026-02-26T03:42:57.567591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'1","last_scrub_stamp":"2026-02-26T03:42:55.715454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:55.715454+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2204,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.565812+0000","last_change":"2026-02-26T03:42:56.009860+0000","last_active":"2026-02-26T03:42:57.565812+0000","last_peered":"2026-02-26T03:42:57.565812+0000","last_clean":"2026-02-26T03:42:57.565812+0000","last_became_active":"2026-02-26T03:25:44.775206+0000","last_became_peered":"2026-02-26T03:25:44.775206+0000","last_unstale":"2026-02-26T03:42:57.565812+0000","last_undegraded":"2026-02-26T03:42:57.565812+0000","last_fullsized":"2026-02-26T03:42:57.565812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:56.009790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:56.009790+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2204,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.566704+0000","last_change":"2026-02-26T03:42:56.010729+0000","last_active":"2026-02-26T03:42:57.566704+0000","last_peered":"2026-02-26T03:42:57.566704+0000","last_clean":"2026-02-26T03:42:57.566704+0000","last_became_active":"2026-02-26T03:25:44.780653+0000","last_became_peered":"2026-02-26T03:25:44.780653+0000","last_unstale":"2026-02-26T03:42:57.566704+0000","last_undegraded":"2026-02-26T03:42:57.566704+0000","last_fullsized":"2026-02-26T03:42:57.566704+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:56.010663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:56.010663+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":2206,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.567824+0000","last_change":"2026-02-26T03:42:55.412663+0000","last_active":"2026-02-26T03:42:57.567824+0000","last_peered":"2026-02-26T03:42:57.567824+0000","last_clean":"2026-02-26T03:42:57.567824+0000","last_became_active":"2026-02-26T03:25:44.770887+0000","last_became_peered":"2026-02-26T03:25:44.770887+0000","last_unstale":"2026-02-26T03:42:57.567824+0000","last_undegraded":"2026-02-26T03:42:57.567824+0000","last_fullsized":"2026-02-26T03:42:57.567824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'2","last_scrub_stamp":"2026-02-26T03:42:55.412586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:55.412586+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2204,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.567591+0000","last_change":"2026-02-26T03:42:55.710506+0000","last_active":"2026-02-26T03:42:57.567591+0000","last_peered":"2026-02-26T03:42:57.567591+0000","last_clean":"2026-02-26T03:42:57.567591+0000","last_became_active":"2026-02-26T03:25:44.764783+0000","last_became_peered":"2026-02-26T03:25:44.764783+0000","last_unstale":"2026-02-26T03:42:57.567591+0000","last_undegraded":"2026-02-26T03:42:57.567591+0000","last_fullsized":"2026-02-26T03:42:57.567591+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:55.710386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:55.710386+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2204,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.568015+0000","last_change":"2026-02-26T03:42:55.714796+0000","last_active":"2026-02-26T03:42:57.568015+0000","last_peered":"2026-02-26T03:42:57.568015+0000","last_clean":"2026-02-26T03:42:57.568015+0000","last_became_active":"2026-02-26T03:25:44.769736+0000","last_became_peered":"2026-02-26T03:25:44.769736+0000","last_unstale":"2026-02-26T03:42:57.568015+0000","last_undegraded":"2026-02-26T03:42:57.568015+0000","last_fullsized":"2026-02-26T03:42:57.568015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:55.714698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:55.714698+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2204,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.567882+0000","last_change":"2026-02-26T03:42:55.712947+0000","last_active":"2026-02-26T03:42:57.567882+0000","last_peered":"2026-02-26T03:42:57.567882+0000","last_clean":"2026-02-26T03:42:57.567882+0000","last_became_active":"2026-02-26T03:25:44.758719+0000","last_became_peered":"2026-02-26T03:25:44.758719+0000","last_unstale":"2026-02-26T03:42:57.567882+0000","last_undegraded":"2026-02-26T03:42:57.567882+0000","last_fullsized":"2026-02-26T03:42:57.567882+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:55.712835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:55.712835+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2204,"reported_epoch":745,"state":"active+clean","last_fresh":"2026-02-26T03:42:57.566667+0000","last_change":"2026-02-26T03:42:55.717037+0000","last_active":"2026-02-26T03:42:57.566667+0000","last_peered":"2026-02-26T03:42:57.566667+0000","last_clean":"2026-02-26T03:42:57.566667+0000","last_became_active":"2026-02-26T03:25:44.759336+0000","last_became_peered":"2026-02-26T03:25:44.759336+0000","last_unstale":"2026-02-26T03:42:57.566667+0000","last_undegraded":"2026-02-26T03:42:57.566667+0000","last_fullsized":"2026-02-26T03:42:57.566667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:55.716982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:55.716982+0000","last_clean_scrub_stamp":"2026-02-26T03:25:43.714753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96631992320,"internally_reserved":0,"allocated":64429232128,"data_stored":64429232128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210362368,"internally_reserved":0,"allocated":21476712448,"data_stored":21476712448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949674014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972540,"kb_used_data":20972540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456244,"statfs":{"total":53687074816,"available":32211193856,"internally_reserved":0,"allocated":21475880960,"data_stored":21475880960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20973352,"kb_used_data":20973352,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455432,"statfs":{"total":53687074816,"available":32210362368,"internally_reserved":0,"allocated":21476712448,"data_stored":21476712448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20973280,"kb_used_data":20973280,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455504,"statfs":{"total":53687074816,"available":32210436096,"internally_reserved":0,"allocated":21476638720,"data_stored":21476638720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32210362368,"internally_reserved":0,"allocated":21476712448,"data_stored":21476712448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32210436096,"internally_reserved":0,"allocated":21476638720,"data_stored":21476638720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32210362368,"internally_reserved":0,"allocated":21476712448,"data_stored":21476712448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211193856,"internally_reserved":0,"allocated":21475880960,"data_stored":21475880960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:43:16.424 DEBUG:teuthology.orchestra.run.trial095:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:43:16.499 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:43:16.494+0000 7f6213dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:16.501 INFO:teuthology.orchestra.run.trial095.stderr:2026-02-26T03:43:16.499+0000 7f6213dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:16.627 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:43:16.628 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:43:16.628 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:43:16.628 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:43:16.696 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:43:16.696 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:43:16.696 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:43:16.754 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:43:16.754 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:43:16.754 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:43:16.831 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:43:16.831 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:43:16.832 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:43:16.832 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:43:16.865 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:43:16.865 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:43:16.866 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:43:16.866 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:43:16.918 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:43:16.918 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:43:16.919 DEBUG:teuthology.orchestra.run.trial095:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(SLOW_OPS\)' | grep -E -v 'slow request' | head -n 1 2026-02-26T03:43:16.984 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:43:16.984 DEBUG:teuthology.orchestra.run.trial095:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:43:17.052 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:43:17.053 DEBUG:teuthology.orchestra.run.trial095:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:43:17.119 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:43:17.120 DEBUG:teuthology.orchestra.run.trial095:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:43:17.192 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:43:17.194 DEBUG:teuthology.misc:Transferring archived files from trial095:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71174/data/mon.a.tgz 2026-02-26T03:43:17.195 DEBUG:teuthology.orchestra.run.trial095:> mktemp 2026-02-26T03:43:17.249 INFO:teuthology.orchestra.run.trial095.stdout:/tmp/tmp.WycFlnLGm0 2026-02-26T03:43:17.249 DEBUG:teuthology.orchestra.run.trial095:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WycFlnLGm0 2026-02-26T03:43:17.345 DEBUG:teuthology.orchestra.run.trial095:> sudo chmod 0666 /tmp/tmp.WycFlnLGm0 2026-02-26T03:43:17.428 DEBUG:teuthology.orchestra.remote:trial095:/tmp/tmp.WycFlnLGm0 is 272KB 2026-02-26T03:43:17.523 DEBUG:teuthology.orchestra.run.trial095:> rm -fr /tmp/tmp.WycFlnLGm0 2026-02-26T03:43:17.540 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:43:17.540 DEBUG:teuthology.orchestra.run.trial095:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:43:17.659 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:43:17.660 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:43:17.663 DEBUG:teuthology.misc:Transferring archived files from trial095:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71174/remote/trial095/crash 2026-02-26T03:43:17.664 DEBUG:teuthology.orchestra.run.trial095:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:43:17.724 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:43:17.724 DEBUG:teuthology.orchestra.run.trial095:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:43:17.790 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38420.log 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38176.log 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38420.log.gz 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42695.log 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38176.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38176.log.gz 2026-02-26T03:43:17.791 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39168.log 2026-02-26T03:43:17.803 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38976.log 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42695.log.gz 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38975.log 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.39168.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39168.log.gz 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42744.log 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38976.log.gz 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr:gzip/var/log/ceph/ceph-client.admin.38975.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38975.log.gz 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42842.log 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42744.log.gz 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43038.log 2026-02-26T03:43:17.804 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gz 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42793.log 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42842.log.gz 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38727.log 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.43038.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43038.log.gz 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42793.log: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.42793.log.gz 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43136.log 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42891.log 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.38727.log.gz 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.43136.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43136.log.gz 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43198.log 2026-02-26T03:43:17.805 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42891.log.gz 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38678.log 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.43198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43245.log 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43198.log.gz 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38678.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38678.log.gz 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37679.log 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.43245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38353.log 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43245.log.gz 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.37679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38829.log 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37679.log.gz 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42940.log 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38353.log.gz 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38828.log 2026-02-26T03:43:17.806 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38829.log.gz 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42940.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.36294.log 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42940.log.gz 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37584.log 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38828.log.gz 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph.tmp-client.admin.36294.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.36294.log.gz 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.37584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38580.log 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37584.log.gz 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38206.log 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38580.log: 86.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38580.log.gz 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:43:17.807 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39119.log 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38206.log.gz 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42989.log 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.39119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38490.log 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39119.log.gz 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.42989.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42989.log.gz 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39443.log 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37910.log 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38490.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38490.log.gz 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.39443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43087.log 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39443.log.gz 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.37910.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37910.log.gz 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.43087.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43087.log.gz 2026-02-26T03:43:17.808 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37957.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.37957.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.37863.log -- replaced with /var/log/ceph/ceph-client.admin.37957.log.gz 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38629.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.37863.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37863.log.gz 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38830.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38629.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38629.log.gz 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38515.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38830.log.gz 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39070.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.38515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37726.log 2026-02-26T03:43:17.809 INFO:teuthology.orchestra.run.trial095.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.38515.log.gz 2026-02-26T03:43:17.810 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.39070.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39070.log.gz 2026-02-26T03:43:17.810 INFO:teuthology.orchestra.run.trial095.stderr:/var/log/ceph/ceph-client.admin.37726.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37726.log.gz 2026-02-26T03:43:17.810 INFO:teuthology.orchestra.run.trial095.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:43:17.852 INFO:teuthology.orchestra.run.trial095.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:43:18.101 INFO:teuthology.orchestra.run.trial095.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T03:43:25.270 INFO:teuthology.orchestra.run.trial095.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:43:27.417 INFO:teuthology.orchestra.run.trial095.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:43:28.992 INFO:teuthology.orchestra.run.trial095.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:43:28.994 INFO:teuthology.orchestra.run.trial095.stderr: 2026-02-26T03:43:28.994 INFO:teuthology.orchestra.run.trial095.stderr:real 0m11.213s 2026-02-26T03:43:28.994 INFO:teuthology.orchestra.run.trial095.stderr:user 0m26.989s 2026-02-26T03:43:28.994 INFO:teuthology.orchestra.run.trial095.stderr:sys 0m1.664s 2026-02-26T03:43:28.995 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:43:28.995 DEBUG:teuthology.misc:Transferring archived files from trial095:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71174/remote/trial095/log 2026-02-26T03:43:28.996 DEBUG:teuthology.orchestra.run.trial095:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:43:31.121 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:43:31.186 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:43:31.198 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:43:31.198 DEBUG:teuthology.orchestra.run.trial095:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:43:31.223 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:43:31.235 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:43:31.235 DEBUG:teuthology.orchestra.run.trial095:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:43:31.279 INFO:teuthology.orchestra.run.trial095.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:43:31.308 INFO:teuthology.orchestra.run.trial095.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:43:31.309 INFO:teuthology.orchestra.run.trial095.stdout:=============================================================================== 2026-02-26T03:43:31.309 INFO:teuthology.orchestra.run.trial095.stdout:^- 23.186.168.132 2 7 375 76 +434us[ +434us] +/- 56ms 2026-02-26T03:43:31.309 INFO:teuthology.orchestra.run.trial095.stdout:^- ntp2.zeus.frumentum.media 2 6 377 22 -106ms[ -106ms] +/- 129ms 2026-02-26T03:43:31.309 INFO:teuthology.orchestra.run.trial095.stdout:^- ntp6.kernfusion.at 2 6 377 24 +314us[ +314us] +/- 46ms 2026-02-26T03:43:31.309 INFO:teuthology.orchestra.run.trial095.stdout:^* chi2.us.ntp.li 2 7 377 89 -191us[ -206us] +/- 12ms 2026-02-26T03:43:31.309 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:43:31.322 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:43:31.322 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:43:31.337 DEBUG:teuthology.orchestra.run.trial095:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:43:31.401 DEBUG:teuthology.orchestra.run.trial095:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:43:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:43:31.428 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:43:31.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:43:31.459 INFO:teuthology.task.internal:Duration was 2072.442442 seconds 2026-02-26T03:43:31.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:43:31.476 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:43:31.476 DEBUG:teuthology.orchestra.run.trial095:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:43:31.512 INFO:teuthology.orchestra.run.trial095.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:43:31.681 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:43:31.682 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial095.front.sepia.ceph.com 2026-02-26T03:43:31.682 DEBUG:teuthology.orchestra.run.trial095:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:43:31.740 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:43:31.740 DEBUG:teuthology.orchestra.run.trial095:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:43:31.841 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:43:31.841 DEBUG:teuthology.orchestra.run.trial095:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:43:31.864 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:43:31.865 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:43:31.865 INFO:teuthology.orchestra.run.trial095.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:43:31.866 INFO:teuthology.orchestra.run.trial095.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:43:31.866 INFO:teuthology.orchestra.run.trial095.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:43:31.875 INFO:teuthology.orchestra.run.trial095.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:43:31.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:43:31.893 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:43:31.893 DEBUG:teuthology.orchestra.run.trial095:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:43:31.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:43:31.959 DEBUG:teuthology.orchestra.run.trial095:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:43:32.005 INFO:teuthology.orchestra.run.trial095.stdout:kernel.core_pattern = core 2026-02-26T03:43:32.023 DEBUG:teuthology.orchestra.run.trial095:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:43:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:43:32.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:43:32.095 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:43:32.097 DEBUG:teuthology.misc:Transferring archived files from trial095:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71174/remote/trial095 2026-02-26T03:43:32.097 DEBUG:teuthology.orchestra.run.trial095:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:43:32.154 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:43:32.154 DEBUG:teuthology.orchestra.run.trial095:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:43:32.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:43:32.219 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:43:32.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:43:32.236 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:43:32.236 DEBUG:teuthology.orchestra.run.trial095:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:43:32.257 INFO:teuthology.orchestra.run.trial095.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:43 /home/ubuntu/cephtest 2026-02-26T03:43:32.258 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:43:32.275 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:43:32.314 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} tasks/rbd_python_api_tests} duration: 2072.442441701889 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-26T03:43:32.314 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:43:32.422 INFO:teuthology.run:pass