2026-02-26T03:07:28.906 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T03:07:28.925 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:07:29.008 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71183 branch: main description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} tasks/rbd_python_api_tests_old_format} email: null first_in_suite: false flavor: debug job_id: '71183' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - slow request sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: extra_system_packages: - python3-pytest flavor: debug sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5117 sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 targets: trial181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUXvXM4NpDN/m5Krs+I785215/eVMa2OjDs4Qu7FLCkWwq77aDcpHVLu1BF+Yn+do/wzzrm4LRAFSS9wuffLss= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 - workunit: clients: client.0: - rbd/test_librbd_python.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_02:22:56 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T03:07:29.008 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa; will attempt to use it 2026-02-26T03:07:29.009 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa/tasks 2026-02-26T03:07:29.009 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T03:07:29.011 INFO:teuthology.task.internal:Checking packages... 2026-02-26T03:07:29.011 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'abffb617683951511fccf3c3cab6e219f2ec83a8' 2026-02-26T03:07:29.011 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T03:07:29.011 INFO:teuthology.packaging:ref: None 2026-02-26T03:07:29.011 INFO:teuthology.packaging:tag: None 2026-02-26T03:07:29.011 INFO:teuthology.packaging:branch: main 2026-02-26T03:07:29.011 INFO:teuthology.packaging:sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:07:29.011 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-26T03:07:29.137 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5593.g63656871 2026-02-26T03:07:29.139 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T03:07:29.150 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T03:07:29.151 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T03:07:29.163 INFO:teuthology.task.internal:Saving configuration 2026-02-26T03:07:29.175 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T03:07:29.187 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T03:07:29.265 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial181.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71183', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 03:04:05.753575', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUXvXM4NpDN/m5Krs+I785215/eVMa2OjDs4Qu7FLCkWwq77aDcpHVLu1BF+Yn+do/wzzrm4LRAFSS9wuffLss='} 2026-02-26T03:07:29.265 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T03:07:29.277 INFO:teuthology.task.internal:roles: ubuntu@trial181.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T03:07:29.277 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T03:07:29.374 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x706295420860>, signals=[15]) 2026-02-26T03:07:29.374 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T03:07:29.383 INFO:teuthology.task.internal:Opening connections... 2026-02-26T03:07:29.383 DEBUG:teuthology.task.internal:connecting to ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:07:29.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:07:29.445 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T03:07:29.457 DEBUG:teuthology.orchestra.run.trial181:> uname -m 2026-02-26T03:07:29.527 INFO:teuthology.orchestra.run.trial181.stdout:x86_64 2026-02-26T03:07:29.527 DEBUG:teuthology.orchestra.run.trial181:> cat /etc/os-release 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:NAME="CentOS Stream" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:VERSION="9" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:ID="centos" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:ID_LIKE="rhel fedora" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:VERSION_ID="9" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:PLATFORM_ID="platform:el9" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:ANSI_COLOR="0;31" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:LOGO="fedora-logo-icon" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:HOME_URL="https://centos.org/" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T03:07:29.585 INFO:teuthology.orchestra.run.trial181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T03:07:29.586 INFO:teuthology.lock.ops:Updating trial181.front.sepia.ceph.com on lock server 2026-02-26T03:07:29.648 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T03:07:29.662 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T03:07:29.672 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T03:07:29.673 DEBUG:teuthology.orchestra.run.trial181:> test '!' -e /home/ubuntu/cephtest 2026-02-26T03:07:29.688 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T03:07:29.699 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T03:07:29.700 DEBUG:teuthology.orchestra.run.trial181:> test -z $(ls -A /var/lib/ceph) 2026-02-26T03:07:29.745 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T03:07:29.833 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T03:07:29.848 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T03:07:29.848 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T03:07:29.849 DEBUG:teuthology.orchestra.run.trial181:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T03:07:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:07:29.864 DEBUG:teuthology.orchestra.run.trial181:> uname -r 2026-02-26T03:07:29.920 INFO:teuthology.orchestra.run.trial181.stdout:5.14.0-681.el9.x86_64 2026-02-26T03:07:29.920 INFO:teuthology.task.kernel:Running kernel on trial181: 5.14.0-681.el9.x86_64 2026-02-26T03:07:29.920 DEBUG:teuthology.orchestra.run.trial181:> sudo yum install -y kernel 2026-02-26T03:08:44.620 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - BaseOS 123 kB/s | 8.9 MB 01:14 2026-02-26T03:08:48.095 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 27 MB 00:02 2026-02-26T03:08:51.772 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 8.0 MB 00:01 2026-02-26T03:08:52.944 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 20 kB 00:00 2026-02-26T03:08:58.918 INFO:teuthology.orchestra.run.trial181.stdout:Extra Packages for Enterprise Linux 3.4 MB/s | 20 MB 00:05 2026-02-26T03:09:02.059 INFO:teuthology.orchestra.run.trial181.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-26T03:09:03.070 INFO:teuthology.orchestra.run.trial181.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T03:09:03.070 INFO:teuthology.orchestra.run.trial181.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T03:09:03.090 INFO:teuthology.orchestra.run.trial181.stdout:Dependencies resolved. 2026-02-26T03:09:03.091 INFO:teuthology.orchestra.run.trial181.stdout:================================================================================ 2026-02-26T03:09:03.091 INFO:teuthology.orchestra.run.trial181.stdout: Package Architecture Version Repository Size 2026-02-26T03:09:03.091 INFO:teuthology.orchestra.run.trial181.stdout:================================================================================ 2026-02-26T03:09:03.091 INFO:teuthology.orchestra.run.trial181.stdout:Installing: 2026-02-26T03:09:03.091 INFO:teuthology.orchestra.run.trial181.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout:Installing dependencies: 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout:Transaction Summary 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout:================================================================================ 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout:Install 4 Packages 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:09:03.092 INFO:teuthology.orchestra.run.trial181.stdout:Total download size: 89 M 2026-02-26T03:09:03.093 INFO:teuthology.orchestra.run.trial181.stdout:Installed size: 135 M 2026-02-26T03:09:03.093 INFO:teuthology.orchestra.run.trial181.stdout:Downloading Packages: 2026-02-26T03:09:04.230 INFO:teuthology.orchestra.run.trial181.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 1.1 MB/s | 901 kB 00:00 2026-02-26T03:09:04.942 INFO:teuthology.orchestra.run.trial181.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 11 MB/s | 17 MB 00:01 2026-02-26T03:09:05.201 INFO:teuthology.orchestra.run.trial181.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 32 MB/s | 31 MB 00:00 2026-02-26T03:09:10.896 INFO:teuthology.orchestra.run.trial181.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 5.3 MB/s | 40 MB 00:07 2026-02-26T03:09:10.898 INFO:teuthology.orchestra.run.trial181.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:09:10.898 INFO:teuthology.orchestra.run.trial181.stdout:Total 11 MB/s | 89 MB 00:07 2026-02-26T03:09:11.091 INFO:teuthology.orchestra.run.trial181.stdout:Running transaction check 2026-02-26T03:09:11.298 INFO:teuthology.orchestra.run.trial181.stdout:Transaction check succeeded. 2026-02-26T03:09:11.298 INFO:teuthology.orchestra.run.trial181.stdout:Running transaction test 2026-02-26T03:09:11.679 INFO:teuthology.orchestra.run.trial181.stdout:Transaction test succeeded. 2026-02-26T03:09:11.679 INFO:teuthology.orchestra.run.trial181.stdout:Running transaction 2026-02-26T03:09:12.789 INFO:teuthology.orchestra.run.trial181.stdout: Preparing : 1/1 2026-02-26T03:09:13.077 INFO:teuthology.orchestra.run.trial181.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:09:13.099 INFO:teuthology.orchestra.run.trial181.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:09:15.661 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:09:15.693 INFO:teuthology.orchestra.run.trial181.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:09:18.942 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:09:18.950 INFO:teuthology.orchestra.run.trial181.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:09:21.999 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:09:39.459 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:09:39.482 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:09:39.557 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:09:39.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:09:39.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:09:39.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:09:39.614 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:09:39.614 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:09:39.614 INFO:teuthology.orchestra.run.trial181.stdout:Installed: 2026-02-26T03:09:39.615 INFO:teuthology.orchestra.run.trial181.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T03:09:39.615 INFO:teuthology.orchestra.run.trial181.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T03:09:39.615 INFO:teuthology.orchestra.run.trial181.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T03:09:39.615 INFO:teuthology.orchestra.run.trial181.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T03:09:39.615 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:09:39.615 INFO:teuthology.orchestra.run.trial181.stdout:Complete! 2026-02-26T03:09:39.676 DEBUG:teuthology.orchestra.run.trial181:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:09:39.705 INFO:teuthology.orchestra.run.trial181.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:09:39.706 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:09:39.706 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T03:09:39.706 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T03:09:39.706 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T03:09:39.706 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T03:09:39.706 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial181.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T03:09:39.706 DEBUG:teuthology.orchestra.run.trial181:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:09:39.771 INFO:teuthology.orchestra.run.trial181.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:09:39.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:09:39.771 DEBUG:teuthology.orchestra.run.trial181:> sudo rpm -qi grub2-tools 2026-02-26T03:09:39.839 INFO:teuthology.orchestra.run.trial181.stdout:Name : grub2-tools 2026-02-26T03:09:39.839 INFO:teuthology.orchestra.run.trial181.stdout:Epoch : 1 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Version : 2.06 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Release : 120.el9 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Architecture: x86_64 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Group : Unspecified 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Size : 9108611 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:License : GPLv3+ 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T03:09:39.840 INFO:teuthology.orchestra.run.trial181.stdout:Packager : builder@centos.org 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:Vendor : CentOS 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:Summary : Support tools for GRUB. 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:Description : 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:hardware devices. 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:09:39.841 INFO:teuthology.orchestra.run.trial181.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T03:09:39.843 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T03:09:39.843 INFO:teuthology.task.kernel:Updating grub on trial181 to boot 5.14.0-686.el9.x86_64 2026-02-26T03:09:39.843 DEBUG:teuthology.orchestra.run.trial181:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T03:09:39.907 INFO:teuthology.orchestra.run.trial181.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T03:09:39.909 DEBUG:teuthology.orchestra.run.trial181:> sudo command -v grubby && echo yes || echo no 2026-02-26T03:09:39.973 INFO:teuthology.orchestra.run.trial181.stdout:/sbin/grubby 2026-02-26T03:09:39.975 INFO:teuthology.orchestra.run.trial181.stdout:yes 2026-02-26T03:09:39.976 DEBUG:teuthology.orchestra.run.trial181:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T03:09:40.039 INFO:teuthology.orchestra.run.trial181.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T03:09:40.040 DEBUG:teuthology.orchestra.run.trial181:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T03:09:40.113 DEBUG:teuthology.orchestra.run.trial181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T03:09:40.322 INFO:teuthology.orchestra.run.trial181.stderr:Generating grub configuration file ... 2026-02-26T03:09:40.388 INFO:teuthology.orchestra.run.trial181.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T03:09:40.395 INFO:teuthology.orchestra.run.trial181.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T03:09:40.646 INFO:teuthology.orchestra.run.trial181.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T03:09:40.653 INFO:teuthology.orchestra.run.trial181.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T03:09:40.687 INFO:teuthology.orchestra.run.trial181.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T03:09:40.693 INFO:teuthology.orchestra.run.trial181.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T03:09:40.725 INFO:teuthology.orchestra.run.trial181.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T03:09:40.732 INFO:teuthology.orchestra.run.trial181.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T03:09:40.756 INFO:teuthology.orchestra.run.trial181.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T03:09:40.762 INFO:teuthology.orchestra.run.trial181.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T03:09:40.849 INFO:teuthology.orchestra.run.trial181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T03:09:40.857 INFO:teuthology.orchestra.run.trial181.stderr:done 2026-02-26T03:09:40.859 DEBUG:teuthology.orchestra.run.trial181:> test -d /sys/firmware/efi 2026-02-26T03:09:40.875 DEBUG:teuthology.orchestra.run.trial181:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T03:09:40.939 INFO:teuthology.orchestra.run.trial181.stdout:/boot/efi/EFI/BOOT 2026-02-26T03:09:40.940 INFO:teuthology.orchestra.run.trial181.stdout:/boot/efi/EFI/centos 2026-02-26T03:09:40.941 DEBUG:teuthology.orchestra.run.trial181:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T03:09:41.007 INFO:teuthology.orchestra.run.trial181.stdout:no 2026-02-26T03:09:41.007 DEBUG:teuthology.orchestra.run.trial181:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T03:09:41.071 INFO:teuthology.orchestra.run.trial181.stdout:yes 2026-02-26T03:09:41.072 DEBUG:teuthology.orchestra.run.trial181:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T03:09:41.137 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial181 2026-02-26T03:09:41.137 DEBUG:teuthology.orchestra.run.trial181:> sudo bash -c ' 2026-02-26T03:09:41.137 DEBUG:teuthology.orchestra.run.trial181:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T03:09:41.138 DEBUG:teuthology.orchestra.run.trial181:> ' 2026-02-26T03:10:11.180 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T03:10:11.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial181.front.sepia.ceph.com' 2026-02-26T03:10:11.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:11:11.243 DEBUG:teuthology.orchestra.remote:timed out 2026-02-26T03:11:20.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial181.front.sepia.ceph.com' 2026-02-26T03:11:20.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:11:20.355 DEBUG:teuthology.orchestra.run.trial181:> true 2026-02-26T03:11:20.507 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial181.front.sepia.ceph.com' 2026-02-26T03:11:20.507 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T03:11:20.507 DEBUG:teuthology.orchestra.run.trial181:> uname -r 2026-02-26T03:11:20.563 INFO:teuthology.orchestra.run.trial181.stdout:5.14.0-686.el9.x86_64 2026-02-26T03:11:20.564 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T03:11:20.564 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T03:11:20.564 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T03:11:21.565 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T03:11:21.565 DEBUG:teuthology.orchestra.run.trial181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T03:11:21.596 INFO:teuthology.orchestra.run.trial181.stdout:ttyS1 2026-02-26T03:11:21.620 DEBUG:teuthology.parallel:result is None 2026-02-26T03:11:21.620 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T03:11:21.632 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T03:11:21.633 DEBUG:teuthology.orchestra.run.trial181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T03:11:21.676 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T03:11:21.689 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T03:11:21.702 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T03:11:21.702 DEBUG:teuthology.orchestra.run.trial181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T03:11:21.739 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T03:11:21.750 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T03:11:21.750 DEBUG:teuthology.orchestra.run.trial181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T03:11:21.797 INFO:teuthology.orchestra.run.trial181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:11:21.807 INFO:teuthology.orchestra.run.trial181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:11:21.809 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T03:11:21.821 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T03:11:21.822 DEBUG:teuthology.orchestra.run.trial181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T03:11:21.873 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T03:11:21.890 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T03:11:21.891 DEBUG:teuthology.orchestra.run.trial181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T03:11:21.930 DEBUG:teuthology.orchestra.run.trial181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:11:22.064 DEBUG:teuthology.orchestra.run.trial181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:11:22.089 DEBUG:teuthology.orchestra.run.trial181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:11:22.233 DEBUG:teuthology.orchestra.run.trial181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:11:22.258 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:11:22.258 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T03:11:22.321 DEBUG:teuthology.orchestra.run.trial181:> sudo service rsyslog restart 2026-02-26T03:11:22.390 INFO:teuthology.orchestra.run.trial181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:11:22.709 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T03:11:22.721 INFO:teuthology.task.internal:Starting timer... 2026-02-26T03:11:22.721 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T03:11:22.739 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T03:11:22.754 DEBUG:teuthology.orchestra.run.trial181:> sudo service auditd rotate 2026-02-26T03:11:22.782 INFO:teuthology.orchestra.run.trial181.stdout:Rotating logs: 2026-02-26T03:11:22.783 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T03:11:22.784 DEBUG:teuthology.orchestra.run.trial181:> /usr/sbin/getenforce 2026-02-26T03:11:22.839 INFO:teuthology.orchestra.run.trial181.stdout:Permissive 2026-02-26T03:11:22.839 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial181.front.sepia.ceph.com': 'permissive'} 2026-02-26T03:11:22.839 DEBUG:teuthology.orchestra.run.trial181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:11:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:11:22.906 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T03:11:22.906 DEBUG:teuthology.orchestra.run.trial181:> sudo /usr/sbin/setenforce permissive 2026-02-26T03:11:22.970 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T03:11:22.983 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T03:11:22.993 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T03:11:22.994 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T03:14:05.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial181.front.sepia.ceph.com')] 2026-02-26T03:14:05.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial181.front.sepia.ceph.com' 2026-02-26T03:14:05.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:14:05.947 DEBUG:teuthology.orchestra.run.trial181:> true 2026-02-26T03:14:06.020 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial181.front.sepia.ceph.com' 2026-02-26T03:14:06.020 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T03:14:06.032 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T03:14:06.032 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T03:14:06.032 DEBUG:teuthology.orchestra.run.trial181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:14:06.086 INFO:teuthology.orchestra.run.trial181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T03:14:06.098 INFO:teuthology.orchestra.run.trial181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T03:14:06.121 INFO:teuthology.orchestra.run.trial181.stderr:sudo: ntpd: command not found 2026-02-26T03:14:06.130 INFO:teuthology.orchestra.run.trial181.stdout:506 Cannot talk to daemon 2026-02-26T03:14:06.142 INFO:teuthology.orchestra.run.trial181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T03:14:06.154 INFO:teuthology.orchestra.run.trial181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T03:14:06.200 INFO:teuthology.orchestra.run.trial181.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:14:06.223 INFO:teuthology.orchestra.run.trial181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:14:06.224 INFO:teuthology.orchestra.run.trial181.stdout:=============================================================================== 2026-02-26T03:14:06.224 INFO:teuthology.orchestra.run.trial181.stdout:^? ntp.silvertree.org 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:14:06.224 INFO:teuthology.orchestra.run.trial181.stdout:^? ntp.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:14:06.224 INFO:teuthology.orchestra.run.trial181.stdout:^? ip229.ip-51-81-226.us 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:14:06.224 INFO:teuthology.orchestra.run.trial181.stdout:^? nyc2.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:14:06.225 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:14:06.239 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:14:06.239 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:14:06.239 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:14:06.239 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:14:06.248 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:14:06.249 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-26T03:14:06.249 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:14:06.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:14:06.406 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:14:06.406 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:14:06.745 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:14:06.746 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:14:06.746 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:14:06.772 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo, python3-pytest on remote rpm x86_64 2026-02-26T03:14:06.773 DEBUG:teuthology.orchestra.run.trial181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:14:06.838 DEBUG:teuthology.orchestra.run.trial181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:14:06.910 DEBUG:teuthology.orchestra.run.trial181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:14:06.974 INFO:teuthology.orchestra.run.trial181.stdout:check_obsoletes = 1 2026-02-26T03:14:06.975 DEBUG:teuthology.orchestra.run.trial181:> sudo yum clean all 2026-02-26T03:14:07.122 INFO:teuthology.orchestra.run.trial181.stdout:41 files removed 2026-02-26T03:14:07.135 DEBUG:teuthology.orchestra.run.trial181:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-osd-crimson ceph-osd-crimson-debuginfo python3-pytest 2026-02-26T03:14:07.989 INFO:teuthology.orchestra.run.trial181.stdout:ceph packages for x86_64 133 kB/s | 94 kB 00:00 2026-02-26T03:14:08.647 INFO:teuthology.orchestra.run.trial181.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-26T03:14:09.242 INFO:teuthology.orchestra.run.trial181.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:14:10.026 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-26T03:14:11.169 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - AppStream 44 MB/s | 27 MB 00:00 2026-02-26T03:14:13.903 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - CRB 23 MB/s | 8.0 MB 00:00 2026-02-26T03:14:15.020 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 20 kB 00:00 2026-02-26T03:14:15.484 INFO:teuthology.orchestra.run.trial181.stdout:Extra Packages for Enterprise Linux 51 MB/s | 20 MB 00:00 2026-02-26T03:14:18.701 INFO:teuthology.orchestra.run.trial181.stdout:lab-extras 2.4 MB/s | 50 kB 00:00 2026-02-26T03:14:19.915 INFO:teuthology.orchestra.run.trial181.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2026-02-26T03:14:19.915 INFO:teuthology.orchestra.run.trial181.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2026-02-26T03:14:19.943 INFO:teuthology.orchestra.run.trial181.stdout:Dependencies resolved. 2026-02-26T03:14:19.948 INFO:teuthology.orchestra.run.trial181.stdout:======================================================================================= 2026-02-26T03:14:19.948 INFO:teuthology.orchestra.run.trial181.stdout: Package Arch Version Repository Size 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout:======================================================================================= 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout:Installing: 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph x86_64 2:20.3.0-5312.gabffb617.el9 ceph 6.5 k 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph-base x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.1 M 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 914 k 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.gabffb617.el9 ceph 149 k 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr x86_64 2:20.3.0-5312.gabffb617.el9 ceph 937 k 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 184 k 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 51 M 2026-02-26T03:14:19.949 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 7.4 M 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 50 k 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 163 k 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.gabffb617.el9 ceph 15 M 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 M 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.gabffb617.el9 ceph 429 M 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-radosgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 11 M 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-test x86_64 2:20.3.0-5312.gabffb617.el9 ceph 126 M 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: ceph-volume noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 301 k 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 1.0 M 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:14:19.950 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 859 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: librados-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 124 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: python3-cephfs x86_64 2:20.3.0-5312.gabffb617.el9 ceph 203 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: python3-pytest noarch 6.2.2-7.el9 appstream 519 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: python3-rados x86_64 2:20.3.0-5312.gabffb617.el9 ceph 324 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: python3-rbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 305 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: python3-rgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 100 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: rbd-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: rbd-mirror x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.6 M 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: rbd-nbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 166 k 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout:Upgrading: 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: librados2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 3.3 M 2026-02-26T03:14:19.951 INFO:teuthology.orchestra.run.trial181.stdout: librbd1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.7 M 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout:Installing dependencies: 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 21 M 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 57 k 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mds x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.0 M 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 297 k 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 4.9 M 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 17 k 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: ceph-selinux x86_64 2:20.3.0-5312.gabffb617.el9 ceph 25 k 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:14:19.952 INFO:teuthology.orchestra.run.trial181.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 26 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: libcephsqlite x86_64 2:20.3.0-5312.gabffb617.el9 ceph 153 k 2026-02-26T03:14:19.953 INFO:teuthology.orchestra.run.trial181.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libradosstriper1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 263 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: librgw2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.6 M 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T03:14:19.954 INFO:teuthology.orchestra.run.trial181.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:14:19.955 INFO:teuthology.orchestra.run.trial181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 45 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 185 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:14:19.956 INFO:teuthology.orchestra.run.trial181.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-iniconfig noarch 1.1.1-7.el9 appstream 17 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:14:19.958 INFO:teuthology.orchestra.run.trial181.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-pluggy noarch 0.13.1-7.el9 appstream 41 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-py noarch 1.10.0-6.el9 appstream 477 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:14:19.959 INFO:teuthology.orchestra.run.trial181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:14:19.960 INFO:teuthology.orchestra.run.trial181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-26T03:14:19.961 INFO:teuthology.orchestra.run.trial181.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout:Installing weak dependencies: 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: ceph-debugsource x86_64 2:20.3.0-5312.gabffb617.el9 ceph 12 M 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 22 k 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout:Transaction Summary 2026-02-26T03:14:19.962 INFO:teuthology.orchestra.run.trial181.stdout:======================================================================================= 2026-02-26T03:14:19.963 INFO:teuthology.orchestra.run.trial181.stdout:Install 156 Packages 2026-02-26T03:14:19.963 INFO:teuthology.orchestra.run.trial181.stdout:Upgrade 2 Packages 2026-02-26T03:14:19.963 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:14:19.963 INFO:teuthology.orchestra.run.trial181.stdout:Total download size: 863 M 2026-02-26T03:14:19.963 INFO:teuthology.orchestra.run.trial181.stdout:Downloading Packages: 2026-02-26T03:14:20.826 INFO:teuthology.orchestra.run.trial181.stdout:(1/158): ceph-20.3.0-5312.gabffb617.el9.x86_64. 23 kB/s | 6.5 kB 00:00 2026-02-26T03:14:27.229 INFO:teuthology.orchestra.run.trial181.stdout:(2/158): ceph-common-20.3.0-5312.gabffb617.el9. 3.1 MB/s | 21 MB 00:06 2026-02-26T03:14:27.706 INFO:teuthology.orchestra.run.trial181.stdout:(3/158): ceph-fuse-20.3.0-5312.gabffb617.el9.x8 1.9 MB/s | 914 kB 00:00 2026-02-26T03:14:27.835 INFO:teuthology.orchestra.run.trial181.stdout:(4/158): ceph-immutable-object-cache-20.3.0-531 1.1 MB/s | 149 kB 00:00 2026-02-26T03:14:28.873 INFO:teuthology.orchestra.run.trial181.stdout:(5/158): ceph-mds-20.3.0-5312.gabffb617.el9.x86 1.9 MB/s | 2.0 MB 00:01 2026-02-26T03:14:29.356 INFO:teuthology.orchestra.run.trial181.stdout:(6/158): ceph-mgr-20.3.0-5312.gabffb617.el9.x86 1.9 MB/s | 937 kB 00:00 2026-02-26T03:14:30.231 INFO:teuthology.orchestra.run.trial181.stdout:(7/158): ceph-base-20.3.0-5312.gabffb617.el9.x8 539 kB/s | 5.1 MB 00:09 2026-02-26T03:14:30.918 INFO:teuthology.orchestra.run.trial181.stdout:(8/158): ceph-osd-20.3.0-5312.gabffb617.el9.x86 238 kB/s | 163 kB 00:00 2026-02-26T03:14:31.707 INFO:teuthology.orchestra.run.trial181.stdout:(9/158): ceph-debugsource-20.3.0-5312.gabffb617 1.1 MB/s | 12 MB 00:10 2026-02-26T03:14:33.229 INFO:teuthology.orchestra.run.trial181.stdout:(10/158): ceph-mon-20.3.0-5312.gabffb617.el9.x8 1.3 MB/s | 4.9 MB 00:03 2026-02-26T03:14:54.685 INFO:teuthology.orchestra.run.trial181.stdout:(11/158): ceph-osd-classic-20.3.0-5312.gabffb61 636 kB/s | 15 MB 00:23 2026-02-26T03:15:07.163 INFO:teuthology.orchestra.run.trial181.stdout:(12/158): ceph-radosgw-20.3.0-5312.gabffb617.el 917 kB/s | 11 MB 00:12 2026-02-26T03:15:07.229 INFO:teuthology.orchestra.run.trial181.stdout:(13/158): ceph-selinux-20.3.0-5312.gabffb617.el 377 kB/s | 25 kB 00:00 2026-02-26T03:16:39.340 INFO:teuthology.orchestra.run.trial181.stdout:(14/158): ceph-osd-crimson-20.3.0-5312.gabffb61 695 kB/s | 87 MB 02:07 2026-02-26T03:16:39.502 INFO:teuthology.orchestra.run.trial181.stdout:(15/158): libcephfs-daemon-20.3.0-5312.gabffb61 229 kB/s | 37 kB 00:00 2026-02-26T03:16:39.642 INFO:teuthology.orchestra.run.trial181.stdout:(16/158): libcephfs-devel-20.3.0-5312.gabffb617 268 kB/s | 37 kB 00:00 2026-02-26T03:16:39.994 INFO:teuthology.orchestra.run.trial181.stdout:(17/158): libcephfs-proxy2-20.3.0-5312.gabffb61 73 kB/s | 26 kB 00:00 2026-02-26T03:16:42.086 INFO:teuthology.orchestra.run.trial181.stdout:(18/158): libcephfs2-20.3.0-5312.gabffb617.el9. 411 kB/s | 859 kB 00:02 2026-02-26T03:16:42.365 INFO:teuthology.orchestra.run.trial181.stdout:(19/158): libcephsqlite-20.3.0-5312.gabffb617.e 550 kB/s | 153 kB 00:00 2026-02-26T03:16:42.704 INFO:teuthology.orchestra.run.trial181.stdout:(20/158): librados-devel-20.3.0-5312.gabffb617. 364 kB/s | 124 kB 00:00 2026-02-26T03:16:43.249 INFO:teuthology.orchestra.run.trial181.stdout:(21/158): libradosstriper1-20.3.0-5312.gabffb61 483 kB/s | 263 kB 00:00 2026-02-26T03:16:51.705 INFO:teuthology.orchestra.run.trial181.stdout:(22/158): librgw2-20.3.0-5312.gabffb617.el9.x86 681 kB/s | 5.6 MB 00:08 2026-02-26T03:16:51.775 INFO:teuthology.orchestra.run.trial181.stdout:(23/158): python3-ceph-argparse-20.3.0-5312.gab 642 kB/s | 45 kB 00:00 2026-02-26T03:16:52.056 INFO:teuthology.orchestra.run.trial181.stdout:(24/158): python3-ceph-common-20.3.0-5312.gabff 657 kB/s | 185 kB 00:00 2026-02-26T03:16:52.334 INFO:teuthology.orchestra.run.trial181.stdout:(25/158): python3-cephfs-20.3.0-5312.gabffb617. 733 kB/s | 203 kB 00:00 2026-02-26T03:16:52.879 INFO:teuthology.orchestra.run.trial181.stdout:(26/158): python3-rados-20.3.0-5312.gabffb617.e 595 kB/s | 324 kB 00:00 2026-02-26T03:16:53.517 INFO:teuthology.orchestra.run.trial181.stdout:(27/158): python3-rbd-20.3.0-5312.gabffb617.el9 478 kB/s | 305 kB 00:00 2026-02-26T03:16:53.765 INFO:teuthology.orchestra.run.trial181.stdout:(28/158): python3-rgw-20.3.0-5312.gabffb617.el9 406 kB/s | 100 kB 00:00 2026-02-26T03:16:53.969 INFO:teuthology.orchestra.run.trial181.stdout:(29/158): rbd-fuse-20.3.0-5312.gabffb617.el9.x8 424 kB/s | 87 kB 00:00 2026-02-26T03:16:59.411 INFO:teuthology.orchestra.run.trial181.stdout:(30/158): rbd-mirror-20.3.0-5312.gabffb617.el9. 488 kB/s | 2.6 MB 00:05 2026-02-26T03:16:59.616 INFO:teuthology.orchestra.run.trial181.stdout:(31/158): rbd-nbd-20.3.0-5312.gabffb617.el9.x86 812 kB/s | 166 kB 00:00 2026-02-26T03:16:59.687 INFO:teuthology.orchestra.run.trial181.stdout:(32/158): ceph-grafana-dashboards-20.3.0-5312.g 805 kB/s | 57 kB 00:00 2026-02-26T03:17:00.122 INFO:teuthology.orchestra.run.trial181.stdout:(33/158): ceph-mgr-cephadm-20.3.0-5312.gabffb61 422 kB/s | 184 kB 00:00 2026-02-26T03:18:23.770 INFO:teuthology.orchestra.run.trial181.stdout:(34/158): ceph-mgr-dashboard-20.3.0-5312.gabffb 620 kB/s | 51 MB 01:23 2026-02-26T03:18:37.100 INFO:teuthology.orchestra.run.trial181.stdout:(35/158): ceph-mgr-diskprediction-local-20.3.0- 567 kB/s | 7.4 MB 00:13 2026-02-26T03:18:37.512 INFO:teuthology.orchestra.run.trial181.stdout:(36/158): ceph-mgr-k8sevents-20.3.0-5312.gabffb 54 kB/s | 22 kB 00:00 2026-02-26T03:18:38.243 INFO:teuthology.orchestra.run.trial181.stdout:(37/158): ceph-mgr-modules-core-20.3.0-5312.gab 406 kB/s | 297 kB 00:00 2026-02-26T03:18:38.378 INFO:teuthology.orchestra.run.trial181.stdout:(38/158): ceph-mgr-rook-20.3.0-5312.gabffb617.e 374 kB/s | 50 kB 00:00 2026-02-26T03:18:38.447 INFO:teuthology.orchestra.run.trial181.stdout:(39/158): ceph-prometheus-alerts-20.3.0-5312.ga 251 kB/s | 17 kB 00:00 2026-02-26T03:18:39.551 INFO:teuthology.orchestra.run.trial181.stdout:(40/158): ceph-test-20.3.0-5312.gabffb617.el9.x 607 kB/s | 126 MB 03:32 2026-02-26T03:18:39.551 INFO:teuthology.orchestra.run.trial181.stdout:(41/158): ceph-volume-20.3.0-5312.gabffb617.el9 273 kB/s | 301 kB 00:01 2026-02-26T03:18:39.667 INFO:teuthology.orchestra.run.trial181.stdout:(42/158): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.0 MB/s | 351 kB 00:00 2026-02-26T03:18:39.751 INFO:teuthology.orchestra.run.trial181.stdout:(43/158): hwloc-libs-2.4.1-6.el9.x86_64.rpm 25 MB/s | 2.1 MB 00:00 2026-02-26T03:18:39.772 INFO:teuthology.orchestra.run.trial181.stdout:(44/158): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.9 MB/s | 40 kB 00:00 2026-02-26T03:18:39.792 INFO:teuthology.orchestra.run.trial181.stdout:(45/158): libatomic-11.5.0-14.el9.x86_64.rpm 1.1 MB/s | 23 kB 00:00 2026-02-26T03:18:39.813 INFO:teuthology.orchestra.run.trial181.stdout:(46/158): libconfig-1.7.2-9.el9.x86_64.rpm 3.4 MB/s | 72 kB 00:00 2026-02-26T03:18:39.836 INFO:teuthology.orchestra.run.trial181.stdout:(47/158): libgfortran-11.5.0-14.el9.x86_64.rpm 33 MB/s | 794 kB 00:00 2026-02-26T03:18:39.858 INFO:teuthology.orchestra.run.trial181.stdout:(48/158): libquadmath-11.5.0-14.el9.x86_64.rpm 8.3 MB/s | 184 kB 00:00 2026-02-26T03:18:39.879 INFO:teuthology.orchestra.run.trial181.stdout:(49/158): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.4 MB/s | 94 kB 00:00 2026-02-26T03:18:39.900 INFO:teuthology.orchestra.run.trial181.stdout:(50/158): mailcap-2.1.49-5.el9.noarch.rpm 1.6 MB/s | 33 kB 00:00 2026-02-26T03:18:39.923 INFO:teuthology.orchestra.run.trial181.stdout:(51/158): nvme-cli-2.16-1.el9.x86_64.rpm 50 MB/s | 1.2 MB 00:00 2026-02-26T03:18:39.944 INFO:teuthology.orchestra.run.trial181.stdout:(52/158): pciutils-3.7.0-7.el9.x86_64.rpm 4.3 MB/s | 93 kB 00:00 2026-02-26T03:18:39.966 INFO:teuthology.orchestra.run.trial181.stdout:(53/158): python3-cffi-1.14.5-5.el9.x86_64.rpm 11 MB/s | 253 kB 00:00 2026-02-26T03:18:39.992 INFO:teuthology.orchestra.run.trial181.stdout:(54/158): python3-cryptography-36.0.1-5.el9.x86 48 MB/s | 1.2 MB 00:00 2026-02-26T03:18:40.014 INFO:teuthology.orchestra.run.trial181.stdout:(55/158): python3-ply-3.11-14.el9.noarch.rpm 4.8 MB/s | 106 kB 00:00 2026-02-26T03:18:40.036 INFO:teuthology.orchestra.run.trial181.stdout:(56/158): python3-pycparser-2.20-6.el9.noarch.r 6.0 MB/s | 135 kB 00:00 2026-02-26T03:18:40.059 INFO:teuthology.orchestra.run.trial181.stdout:(57/158): python3-pyparsing-2.4.7-9.el9.noarch. 6.6 MB/s | 150 kB 00:00 2026-02-26T03:18:40.081 INFO:teuthology.orchestra.run.trial181.stdout:(58/158): python3-requests-2.25.1-10.el9.noarch 5.5 MB/s | 126 kB 00:00 2026-02-26T03:18:40.104 INFO:teuthology.orchestra.run.trial181.stdout:(59/158): python3-urllib3-1.26.5-7.el9.noarch.r 9.3 MB/s | 218 kB 00:00 2026-02-26T03:18:40.127 INFO:teuthology.orchestra.run.trial181.stdout:(60/158): unzip-6.0-59.el9.x86_64.rpm 7.7 MB/s | 182 kB 00:00 2026-02-26T03:18:40.149 INFO:teuthology.orchestra.run.trial181.stdout:(61/158): zip-3.0-35.el9.x86_64.rpm 12 MB/s | 266 kB 00:00 2026-02-26T03:18:40.867 INFO:teuthology.orchestra.run.trial181.stdout:(62/158): boost-program-options-1.75.0-13.el9.x 145 kB/s | 104 kB 00:00 2026-02-26T03:18:41.026 INFO:teuthology.orchestra.run.trial181.stdout:(63/158): flexiblas-3.0.4-9.el9.x86_64.rpm 186 kB/s | 30 kB 00:00 2026-02-26T03:18:43.273 INFO:teuthology.orchestra.run.trial181.stdout:(64/158): cephadm-20.3.0-5312.gabffb617.el9.noa 279 kB/s | 1.0 MB 00:03 2026-02-26T03:18:43.675 INFO:teuthology.orchestra.run.trial181.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:18:44.312 INFO:teuthology.orchestra.run.trial181.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:18:44.355 INFO:teuthology.orchestra.run.trial181.stdout:(65/158): flexiblas-openblas-openmp-3.0.4-9.el9 14 kB/s | 15 kB 00:01 2026-02-26T03:18:44.568 INFO:teuthology.orchestra.run.trial181.stdout:(66/158): flexiblas-netlib-3.0.4-9.el9.x86_64.r 864 kB/s | 3.0 MB 00:03 2026-02-26T03:18:44.728 INFO:teuthology.orchestra.run.trial181.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:18:44.750 INFO:teuthology.orchestra.run.trial181.stdout:[MIRROR] libasan8-15.2.1-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.2.1-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:18:45.364 INFO:teuthology.orchestra.run.trial181.stdout:(67/158): libnbd-1.20.3-4.el9.x86_64.rpm 206 kB/s | 164 kB 00:00 2026-02-26T03:18:45.679 INFO:teuthology.orchestra.run.trial181.stdout:(68/158): libpmemobj-1.12.1-1.el9.x86_64.rpm 509 kB/s | 160 kB 00:00 2026-02-26T03:18:45.771 INFO:teuthology.orchestra.run.trial181.stdout:(69/158): libasan8-15.2.1-7.el9.x86_64.rpm 371 kB/s | 525 kB 00:01 2026-02-26T03:18:45.837 INFO:teuthology.orchestra.run.trial181.stdout:(70/158): librabbitmq-0.11.0-7.el9.x86_64.rpm 287 kB/s | 45 kB 00:00 2026-02-26T03:18:46.072 INFO:teuthology.orchestra.run.trial181.stdout:[MIRROR] libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:18:46.477 INFO:teuthology.orchestra.run.trial181.stdout:[MIRROR] libstoragemgmt-1.10.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libstoragemgmt-1.10.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:18:46.541 INFO:teuthology.orchestra.run.trial181.stdout:(71/158): libstoragemgmt-1.10.1-1.el9.x86_64.rp 350 kB/s | 246 kB 00:00 2026-02-26T03:18:46.650 INFO:teuthology.orchestra.run.trial181.stdout:(72/158): librdkafka-1.6.1-102.el9.x86_64.rpm 753 kB/s | 662 kB 00:00 2026-02-26T03:18:46.810 INFO:teuthology.orchestra.run.trial181.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-26T03:18:47.248 INFO:teuthology.orchestra.run.trial181.stdout:(73/158): libubsan-11.5.0-14.el9.x86_64.rpm 249 kB/s | 176 kB 00:00 2026-02-26T03:18:47.715 INFO:teuthology.orchestra.run.trial181.stdout:(74/158): lttng-ust-2.12.0-6.el9.x86_64.rpm 625 kB/s | 292 kB 00:00 2026-02-26T03:18:48.029 INFO:teuthology.orchestra.run.trial181.stdout:(75/158): lua-5.4.4-4.el9.x86_64.rpm 600 kB/s | 188 kB 00:00 2026-02-26T03:18:48.032 INFO:teuthology.orchestra.run.trial181.stdout:(76/158): libxslt-1.1.34-12.el9.x86_64.rpm 169 kB/s | 233 kB 00:01 2026-02-26T03:18:48.187 INFO:teuthology.orchestra.run.trial181.stdout:(77/158): openblas-0.3.29-1.el9.x86_64.rpm 266 kB/s | 42 kB 00:00 2026-02-26T03:18:49.431 INFO:teuthology.orchestra.run.trial181.stdout:(78/158): protobuf-3.14.0-17.el9.x86_64.rpm 827 kB/s | 1.0 MB 00:01 2026-02-26T03:18:54.091 INFO:teuthology.orchestra.run.trial181.stdout:(79/158): python3-babel-2.9.1-2.el9.noarch.rpm 1.3 MB/s | 6.0 MB 00:04 2026-02-26T03:18:54.251 INFO:teuthology.orchestra.run.trial181.stdout:(80/158): python3-devel-3.9.25-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2026-02-26T03:18:54.409 INFO:teuthology.orchestra.run.trial181.stdout:(81/158): python3-iniconfig-1.1.1-7.el9.noarch. 110 kB/s | 17 kB 00:00 2026-02-26T03:18:54.568 INFO:teuthology.orchestra.run.trial181.stdout:(82/158): python3-jinja2-2.11.3-8.el9.noarch.rp 1.5 MB/s | 249 kB 00:00 2026-02-26T03:18:54.698 INFO:teuthology.orchestra.run.trial181.stdout:(83/158): openblas-openmp-0.3.29-1.el9.x86_64.r 812 kB/s | 5.3 MB 00:06 2026-02-26T03:18:54.727 INFO:teuthology.orchestra.run.trial181.stdout:(84/158): python3-jmespath-1.0.1-1.el9.noarch.r 300 kB/s | 48 kB 00:00 2026-02-26T03:18:54.857 INFO:teuthology.orchestra.run.trial181.stdout:(85/158): python3-libstoragemgmt-1.10.1-1.el9.x 1.1 MB/s | 177 kB 00:00 2026-02-26T03:18:54.969 INFO:teuthology.orchestra.run.trial181.stdout:(86/158): python3-lxml-4.6.5-3.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2026-02-26T03:18:55.017 INFO:teuthology.orchestra.run.trial181.stdout:(87/158): python3-markupsafe-1.1.1-12.el9.x86_6 218 kB/s | 35 kB 00:00 2026-02-26T03:18:55.454 INFO:teuthology.orchestra.run.trial181.stdout:(88/158): python3-numpy-1.23.5-2.el9.x86_64.rpm 13 MB/s | 6.1 MB 00:00 2026-02-26T03:18:55.569 INFO:teuthology.orchestra.run.trial181.stdout:(89/158): python3-numpy-f2py-1.23.5-2.el9.x86_6 801 kB/s | 442 kB 00:00 2026-02-26T03:18:55.612 INFO:teuthology.orchestra.run.trial181.stdout:(90/158): python3-packaging-20.9-5.el9.noarch.r 487 kB/s | 77 kB 00:00 2026-02-26T03:18:55.727 INFO:teuthology.orchestra.run.trial181.stdout:(91/158): python3-pluggy-0.13.1-7.el9.noarch.rp 262 kB/s | 41 kB 00:00 2026-02-26T03:18:55.888 INFO:teuthology.orchestra.run.trial181.stdout:(92/158): python3-py-1.10.0-6.el9.noarch.rpm 2.9 MB/s | 477 kB 00:00 2026-02-26T03:18:55.928 INFO:teuthology.orchestra.run.trial181.stdout:(93/158): python3-protobuf-3.14.0-17.el9.noarch 846 kB/s | 267 kB 00:00 2026-02-26T03:18:56.046 INFO:teuthology.orchestra.run.trial181.stdout:(94/158): python3-pyasn1-0.4.8-7.el9.noarch.rpm 991 kB/s | 157 kB 00:00 2026-02-26T03:18:56.207 INFO:teuthology.orchestra.run.trial181.stdout:(95/158): python3-pytest-6.2.2-7.el9.noarch.rpm 3.2 MB/s | 519 kB 00:00 2026-02-26T03:18:56.244 INFO:teuthology.orchestra.run.trial181.stdout:(96/158): python3-pyasn1-modules-0.4.8-7.el9.no 877 kB/s | 277 kB 00:00 2026-02-26T03:18:56.366 INFO:teuthology.orchestra.run.trial181.stdout:(97/158): python3-requests-oauthlib-1.3.0-12.el 337 kB/s | 54 kB 00:00 2026-02-26T03:18:56.525 INFO:teuthology.orchestra.run.trial181.stdout:(98/158): python3-toml-0.10.2-6.el9.noarch.rpm 263 kB/s | 42 kB 00:00 2026-02-26T03:18:56.683 INFO:teuthology.orchestra.run.trial181.stdout:(99/158): qatlib-25.08.0-2.el9.x86_64.rpm 1.5 MB/s | 240 kB 00:00 2026-02-26T03:18:56.841 INFO:teuthology.orchestra.run.trial181.stdout:(100/158): qatlib-service-25.08.0-2.el9.x86_64. 235 kB/s | 37 kB 00:00 2026-02-26T03:18:56.999 INFO:teuthology.orchestra.run.trial181.stdout:(101/158): qatzip-libs-1.3.1-1.el9.x86_64.rpm 420 kB/s | 66 kB 00:00 2026-02-26T03:18:57.158 INFO:teuthology.orchestra.run.trial181.stdout:(102/158): socat-1.7.4.1-8.el9.x86_64.rpm 1.9 MB/s | 303 kB 00:00 2026-02-26T03:18:57.317 INFO:teuthology.orchestra.run.trial181.stdout:(103/158): xmlsec1-1.2.29-13.el9.x86_64.rpm 1.2 MB/s | 189 kB 00:00 2026-02-26T03:18:57.475 INFO:teuthology.orchestra.run.trial181.stdout:(104/158): xmlsec1-openssl-1.2.29-13.el9.x86_64 571 kB/s | 90 kB 00:00 2026-02-26T03:18:57.633 INFO:teuthology.orchestra.run.trial181.stdout:(105/158): xmlstarlet-1.6.1-20.el9.x86_64.rpm 402 kB/s | 64 kB 00:00 2026-02-26T03:18:57.715 INFO:teuthology.orchestra.run.trial181.stdout:(106/158): lua-devel-5.4.4-4.el9.x86_64.rpm 275 kB/s | 22 kB 00:00 2026-02-26T03:18:57.881 INFO:teuthology.orchestra.run.trial181.stdout:(107/158): protobuf-compiler-3.14.0-17.el9.x86_ 5.1 MB/s | 862 kB 00:00 2026-02-26T03:18:58.406 INFO:teuthology.orchestra.run.trial181.stdout:(108/158): abseil-cpp-20211102.0-4.el9.x86_64.r 1.0 MB/s | 551 kB 00:00 2026-02-26T03:18:58.490 INFO:teuthology.orchestra.run.trial181.stdout:(109/158): gperftools-libs-2.9.1-3.el9.x86_64.r 3.6 MB/s | 308 kB 00:00 2026-02-26T03:18:58.551 INFO:teuthology.orchestra.run.trial181.stdout:(110/158): grpc-data-1.46.7-10.el9.noarch.rpm 316 kB/s | 19 kB 00:00 2026-02-26T03:18:58.812 INFO:teuthology.orchestra.run.trial181.stdout:(111/158): libarrow-9.0.0-15.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2026-02-26T03:18:58.873 INFO:teuthology.orchestra.run.trial181.stdout:(112/158): libarrow-doc-9.0.0-15.el9.noarch.rpm 406 kB/s | 25 kB 00:00 2026-02-26T03:18:58.935 INFO:teuthology.orchestra.run.trial181.stdout:(113/158): liboath-2.6.12-1.el9.x86_64.rpm 801 kB/s | 49 kB 00:00 2026-02-26T03:18:58.996 INFO:teuthology.orchestra.run.trial181.stdout:(114/158): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2026-02-26T03:18:59.058 INFO:teuthology.orchestra.run.trial181.stdout:(115/158): luarocks-3.9.2-5.el9.noarch.rpm 2.4 MB/s | 151 kB 00:00 2026-02-26T03:18:59.127 INFO:teuthology.orchestra.run.trial181.stdout:(116/158): parquet-libs-9.0.0-15.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2026-02-26T03:18:59.193 INFO:teuthology.orchestra.run.trial181.stdout:(117/158): python3-asyncssh-2.13.2-5.el9.noarch 8.1 MB/s | 548 kB 00:00 2026-02-26T03:18:59.254 INFO:teuthology.orchestra.run.trial181.stdout:(118/158): python3-autocommand-2.2.2-8.el9.noar 483 kB/s | 29 kB 00:00 2026-02-26T03:18:59.316 INFO:teuthology.orchestra.run.trial181.stdout:(119/158): python3-backports-tarfile-1.2.0-1.el 983 kB/s | 60 kB 00:00 2026-02-26T03:18:59.377 INFO:teuthology.orchestra.run.trial181.stdout:(120/158): python3-bcrypt-3.2.2-1.el9.x86_64.rp 711 kB/s | 43 kB 00:00 2026-02-26T03:18:59.438 INFO:teuthology.orchestra.run.trial181.stdout:(121/158): python3-cachetools-4.2.4-1.el9.noarc 527 kB/s | 32 kB 00:00 2026-02-26T03:18:59.500 INFO:teuthology.orchestra.run.trial181.stdout:(122/158): python3-certifi-2023.05.07-4.el9.noa 229 kB/s | 14 kB 00:00 2026-02-26T03:18:59.562 INFO:teuthology.orchestra.run.trial181.stdout:(123/158): python3-cheroot-10.0.1-4.el9.noarch. 2.7 MB/s | 173 kB 00:00 2026-02-26T03:18:59.627 INFO:teuthology.orchestra.run.trial181.stdout:(124/158): python3-cherrypy-18.6.1-2.el9.noarch 5.4 MB/s | 358 kB 00:00 2026-02-26T03:18:59.690 INFO:teuthology.orchestra.run.trial181.stdout:(125/158): python3-google-auth-2.45.0-1.el9.noa 3.9 MB/s | 254 kB 00:00 2026-02-26T03:18:59.772 INFO:teuthology.orchestra.run.trial181.stdout:(126/158): python3-grpcio-1.46.7-10.el9.x86_64. 25 MB/s | 2.0 MB 00:00 2026-02-26T03:18:59.834 INFO:teuthology.orchestra.run.trial181.stdout:(127/158): python3-grpcio-tools-1.46.7-10.el9.x 2.3 MB/s | 144 kB 00:00 2026-02-26T03:18:59.896 INFO:teuthology.orchestra.run.trial181.stdout:(128/158): python3-influxdb-5.3.1-1.el9.noarch. 2.2 MB/s | 139 kB 00:00 2026-02-26T03:18:59.957 INFO:teuthology.orchestra.run.trial181.stdout:(129/158): python3-isodate-0.6.1-3.el9.noarch.r 917 kB/s | 56 kB 00:00 2026-02-26T03:19:00.018 INFO:teuthology.orchestra.run.trial181.stdout:(130/158): python3-jaraco-8.2.1-3.el9.noarch.rp 175 kB/s | 11 kB 00:00 2026-02-26T03:19:00.079 INFO:teuthology.orchestra.run.trial181.stdout:(131/158): python3-jaraco-classes-3.2.1-5.el9.n 290 kB/s | 18 kB 00:00 2026-02-26T03:19:00.140 INFO:teuthology.orchestra.run.trial181.stdout:(132/158): python3-jaraco-collections-3.0.0-8.e 380 kB/s | 23 kB 00:00 2026-02-26T03:19:00.201 INFO:teuthology.orchestra.run.trial181.stdout:(133/158): python3-jaraco-context-6.0.1-3.el9.n 322 kB/s | 20 kB 00:00 2026-02-26T03:19:00.269 INFO:teuthology.orchestra.run.trial181.stdout:(134/158): python3-scipy-1.9.3-2.el9.x86_64.rpm 4.8 MB/s | 19 MB 00:04 2026-02-26T03:19:00.270 INFO:teuthology.orchestra.run.trial181.stdout:(135/158): python3-jaraco-functools-3.5.0-2.el9 284 kB/s | 19 kB 00:00 2026-02-26T03:19:00.341 INFO:teuthology.orchestra.run.trial181.stdout:(136/158): python3-kubernetes-26.1.0-3.el9.noar 14 MB/s | 1.0 MB 00:00 2026-02-26T03:19:00.402 INFO:teuthology.orchestra.run.trial181.stdout:(137/158): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2026-02-26T03:19:00.463 INFO:teuthology.orchestra.run.trial181.stdout:(138/158): python3-msgpack-1.0.3-2.el9.x86_64.r 1.4 MB/s | 86 kB 00:00 2026-02-26T03:19:00.525 INFO:teuthology.orchestra.run.trial181.stdout:(139/158): python3-natsort-7.1.1-5.el9.noarch.r 938 kB/s | 58 kB 00:00 2026-02-26T03:19:00.530 INFO:teuthology.orchestra.run.trial181.stdout:(140/158): python3-jaraco-text-4.0.0-2.el9.noar 101 kB/s | 26 kB 00:00 2026-02-26T03:19:00.586 INFO:teuthology.orchestra.run.trial181.stdout:(141/158): python3-portend-3.1.0-2.el9.noarch.r 269 kB/s | 16 kB 00:00 2026-02-26T03:19:00.647 INFO:teuthology.orchestra.run.trial181.stdout:(142/158): python3-repoze-lru-0.7-16.el9.noarch 504 kB/s | 31 kB 00:00 2026-02-26T03:19:00.700 INFO:teuthology.orchestra.run.trial181.stdout:(143/158): python3-pyOpenSSL-21.0.0-1.el9.noarc 533 kB/s | 90 kB 00:00 2026-02-26T03:19:00.710 INFO:teuthology.orchestra.run.trial181.stdout:(144/158): python3-routes-2.5.1-5.el9.noarch.rp 2.9 MB/s | 188 kB 00:00 2026-02-26T03:19:00.772 INFO:teuthology.orchestra.run.trial181.stdout:(145/158): python3-saml-1.16.0-1.el9.noarch.rpm 2.0 MB/s | 125 kB 00:00 2026-02-26T03:19:00.787 INFO:teuthology.orchestra.run.trial181.stdout:(146/158): python3-rsa-4.9-2.el9.noarch.rpm 678 kB/s | 59 kB 00:00 2026-02-26T03:19:00.833 INFO:teuthology.orchestra.run.trial181.stdout:(147/158): python3-tempora-5.0.0-2.el9.noarch.r 585 kB/s | 36 kB 00:00 2026-02-26T03:19:00.886 INFO:teuthology.orchestra.run.trial181.stdout:(148/158): python3-typing-extensions-4.15.0-1.e 874 kB/s | 86 kB 00:00 2026-02-26T03:19:00.895 INFO:teuthology.orchestra.run.trial181.stdout:(149/158): python3-websocket-client-1.2.3-2.el9 1.4 MB/s | 90 kB 00:00 2026-02-26T03:19:00.956 INFO:teuthology.orchestra.run.trial181.stdout:(150/158): python3-xmltodict-0.12.0-15.el9.noar 363 kB/s | 22 kB 00:00 2026-02-26T03:19:00.966 INFO:teuthology.orchestra.run.trial181.stdout:(151/158): python3-xmlsec-1.3.13-1.el9.x86_64.r 605 kB/s | 48 kB 00:00 2026-02-26T03:19:01.017 INFO:teuthology.orchestra.run.trial181.stdout:(152/158): python3-zc-lockfile-2.0-10.el9.noarc 327 kB/s | 20 kB 00:00 2026-02-26T03:19:01.095 INFO:teuthology.orchestra.run.trial181.stdout:(153/158): thrift-0.15.0-4.el9.x86_64.rpm 20 MB/s | 1.6 MB 00:00 2026-02-26T03:19:01.102 INFO:teuthology.orchestra.run.trial181.stdout:(154/158): re2-20211101-20.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2026-02-26T03:19:01.156 INFO:teuthology.orchestra.run.trial181.stdout:(155/158): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.9 MB/s | 122 kB 00:00 2026-02-26T03:19:07.797 INFO:teuthology.orchestra.run.trial181.stdout:(156/158): librados2-20.3.0-5312.gabffb617.el9. 502 kB/s | 3.3 MB 00:06 2026-02-26T03:19:08.798 INFO:teuthology.orchestra.run.trial181.stdout:(157/158): librbd1-20.3.0-5312.gabffb617.el9.x8 360 kB/s | 2.7 MB 00:07 2026-02-26T03:23:27.808 INFO:teuthology.orchestra.run.trial181.stdout:(158/158): ceph-osd-crimson-debuginfo-20.3.0-53 821 kB/s | 429 MB 08:54 2026-02-26T03:23:27.812 INFO:teuthology.orchestra.run.trial181.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:23:27.812 INFO:teuthology.orchestra.run.trial181.stdout:Total 1.6 MB/s | 863 MB 09:07 2026-02-26T03:23:28.353 INFO:teuthology.orchestra.run.trial181.stdout:Running transaction check 2026-02-26T03:23:28.406 INFO:teuthology.orchestra.run.trial181.stdout:Transaction check succeeded. 2026-02-26T03:23:28.406 INFO:teuthology.orchestra.run.trial181.stdout:Running transaction test 2026-02-26T03:23:31.155 INFO:teuthology.orchestra.run.trial181.stdout:Transaction test succeeded. 2026-02-26T03:23:31.155 INFO:teuthology.orchestra.run.trial181.stdout:Running transaction 2026-02-26T03:23:33.971 INFO:teuthology.orchestra.run.trial181.stdout: Preparing : 1/1 2026-02-26T03:23:33.974 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/160 2026-02-26T03:23:33.982 INFO:teuthology.orchestra.run.trial181.stdout: Installing : liboath-2.6.12-1.el9.x86_64 2/160 2026-02-26T03:23:33.993 INFO:teuthology.orchestra.run.trial181.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/160 2026-02-26T03:23:34.137 INFO:teuthology.orchestra.run.trial181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/160 2026-02-26T03:23:34.138 INFO:teuthology.orchestra.run.trial181.stdout: Upgrading : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 5/160 2026-02-26T03:23:34.188 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 5/160 2026-02-26T03:23:34.189 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 6/160 2026-02-26T03:23:34.200 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 6/160 2026-02-26T03:23:34.202 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86 7/160 2026-02-26T03:23:34.202 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86 8/160 2026-02-26T03:23:34.230 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86 8/160 2026-02-26T03:23:34.235 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 9/160 2026-02-26T03:23:34.243 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 10/160 2026-02-26T03:23:34.247 INFO:teuthology.orchestra.run.trial181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 11/160 2026-02-26T03:23:34.250 INFO:teuthology.orchestra.run.trial181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 12/160 2026-02-26T03:23:34.251 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 13/160 2026-02-26T03:23:34.256 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 14/160 2026-02-26T03:23:34.372 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 15/160 2026-02-26T03:23:34.372 INFO:teuthology.orchestra.run.trial181.stdout: Upgrading : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 16/160 2026-02-26T03:23:34.414 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 16/160 2026-02-26T03:23:34.420 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 17/160 2026-02-26T03:23:34.430 INFO:teuthology.orchestra.run.trial181.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 18/160 2026-02-26T03:23:34.430 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 19/160 2026-02-26T03:23:34.453 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 19/160 2026-02-26T03:23:34.454 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86 20/160 2026-02-26T03:23:34.466 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86 20/160 2026-02-26T03:23:34.470 INFO:teuthology.orchestra.run.trial181.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 21/160 2026-02-26T03:23:34.506 INFO:teuthology.orchestra.run.trial181.stdout: Installing : re2-1:20211101-20.el9.x86_64 22/160 2026-02-26T03:23:34.511 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 23/160 2026-02-26T03:23:34.515 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 24/160 2026-02-26T03:23:34.536 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 25/160 2026-02-26T03:23:34.542 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 26/160 2026-02-26T03:23:34.546 INFO:teuthology.orchestra.run.trial181.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/160 2026-02-26T03:23:34.551 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 28/160 2026-02-26T03:23:34.551 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 29/160 2026-02-26T03:23:34.562 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 29/160 2026-02-26T03:23:34.567 INFO:teuthology.orchestra.run.trial181.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 30/160 2026-02-26T03:23:34.591 INFO:teuthology.orchestra.run.trial181.stdout: Installing : unzip-6.0-59.el9.x86_64 31/160 2026-02-26T03:23:34.603 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 32/160 2026-02-26T03:23:34.609 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 33/160 2026-02-26T03:23:34.613 INFO:teuthology.orchestra.run.trial181.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 34/160 2026-02-26T03:23:34.619 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 35/160 2026-02-26T03:23:34.621 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 36/160 2026-02-26T03:23:34.622 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 37/160 2026-02-26T03:23:34.632 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 38/160 2026-02-26T03:23:34.667 INFO:teuthology.orchestra.run.trial181.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 39/160 2026-02-26T03:23:34.671 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-ceph-common-2:20.3.0-5312.gabffb617.el9. 40/160 2026-02-26T03:23:34.682 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el 41/160 2026-02-26T03:23:34.693 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_6 42/160 2026-02-26T03:23:34.699 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 43/160 2026-02-26T03:23:34.723 INFO:teuthology.orchestra.run.trial181.stdout: Installing : zip-3.0-35.el9.x86_64 44/160 2026-02-26T03:23:34.727 INFO:teuthology.orchestra.run.trial181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 45/160 2026-02-26T03:23:34.777 INFO:teuthology.orchestra.run.trial181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 46/160 2026-02-26T03:23:34.790 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 47/160 2026-02-26T03:23:34.797 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/160 2026-02-26T03:23:34.800 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/160 2026-02-26T03:23:34.802 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/160 2026-02-26T03:23:34.807 INFO:teuthology.orchestra.run.trial181.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 51/160 2026-02-26T03:23:34.823 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 52/160 2026-02-26T03:23:34.828 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 53/160 2026-02-26T03:23:34.833 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/160 2026-02-26T03:23:34.840 INFO:teuthology.orchestra.run.trial181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 55/160 2026-02-26T03:23:34.847 INFO:teuthology.orchestra.run.trial181.stdout: Installing : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_6 56/160 2026-02-26T03:23:34.859 INFO:teuthology.orchestra.run.trial181.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 57/160 2026-02-26T03:23:34.864 INFO:teuthology.orchestra.run.trial181.stdout: Installing : thrift-0.15.0-4.el9.x86_64 58/160 2026-02-26T03:23:34.867 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 59/160 2026-02-26T03:23:34.882 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 60/160 2026-02-26T03:23:34.886 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 61/160 2026-02-26T03:23:34.891 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 62/160 2026-02-26T03:23:34.902 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 63/160 2026-02-26T03:23:34.911 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 64/160 2026-02-26T03:23:34.918 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 65/160 2026-02-26T03:23:34.937 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 66/160 2026-02-26T03:23:34.948 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 67/160 2026-02-26T03:23:34.964 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 68/160 2026-02-26T03:23:34.968 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 69/160 2026-02-26T03:23:34.977 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 70/160 2026-02-26T03:23:35.017 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 71/160 2026-02-26T03:23:35.341 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 72/160 2026-02-26T03:23:35.351 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 73/160 2026-02-26T03:23:35.355 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 74/160 2026-02-26T03:23:35.361 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 75/160 2026-02-26T03:23:35.367 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 76/160 2026-02-26T03:23:35.370 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 77/160 2026-02-26T03:23:35.375 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 78/160 2026-02-26T03:23:35.381 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 79/160 2026-02-26T03:23:35.384 INFO:teuthology.orchestra.run.trial181.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 80/160 2026-02-26T03:23:35.385 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 81/160 2026-02-26T03:23:35.414 INFO:teuthology.orchestra.run.trial181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 82/160 2026-02-26T03:23:35.459 INFO:teuthology.orchestra.run.trial181.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 83/160 2026-02-26T03:23:35.468 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 84/160 2026-02-26T03:23:35.515 INFO:teuthology.orchestra.run.trial181.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 85/160 2026-02-26T03:23:35.544 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-py-1.10.0-6.el9.noarch 86/160 2026-02-26T03:23:35.554 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 87/160 2026-02-26T03:23:35.561 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 88/160 2026-02-26T03:23:35.566 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-pluggy-0.13.1-7.el9.noarch 89/160 2026-02-26T03:23:35.573 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 90/160 2026-02-26T03:23:35.577 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 91/160 2026-02-26T03:23:35.612 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-iniconfig-1.1.1-7.el9.noarch 92/160 2026-02-26T03:23:35.836 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 93/160 2026-02-26T03:23:35.859 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 94/160 2026-02-26T03:23:35.863 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 95/160 2026-02-26T03:23:35.922 INFO:teuthology.orchestra.run.trial181.stdout: Installing : openblas-0.3.29-1.el9.x86_64 96/160 2026-02-26T03:23:35.924 INFO:teuthology.orchestra.run.trial181.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 97/160 2026-02-26T03:23:35.946 INFO:teuthology.orchestra.run.trial181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 98/160 2026-02-26T03:23:36.271 INFO:teuthology.orchestra.run.trial181.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 99/160 2026-02-26T03:23:36.345 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 100/160 2026-02-26T03:23:37.012 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 101/160 2026-02-26T03:23:37.026 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 102/160 2026-02-26T03:23:37.032 INFO:teuthology.orchestra.run.trial181.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 103/160 2026-02-26T03:23:37.282 INFO:teuthology.orchestra.run.trial181.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 104/160 2026-02-26T03:23:37.283 INFO:teuthology.orchestra.run.trial181.stdout: Installing : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 105/160 2026-02-26T03:23:37.301 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 105/160 2026-02-26T03:23:37.302 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 106/160 2026-02-26T03:23:38.260 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 107/160 2026-02-26T03:23:38.262 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 107/160 2026-02-26T03:23:38.281 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 107/160 2026-02-26T03:23:38.292 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 108/160 2026-02-26T03:23:38.299 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 109/160 2026-02-26T03:23:38.312 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-ply-3.11-14.el9.noarch 110/160 2026-02-26T03:23:38.328 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 111/160 2026-02-26T03:23:38.403 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 112/160 2026-02-26T03:23:38.411 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 113/160 2026-02-26T03:23:38.435 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 114/160 2026-02-26T03:23:38.464 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 115/160 2026-02-26T03:23:38.514 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 116/160 2026-02-26T03:23:38.520 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 117/160 2026-02-26T03:23:38.523 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 118/160 2026-02-26T03:23:38.528 INFO:teuthology.orchestra.run.trial181.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 119/160 2026-02-26T03:23:38.530 INFO:teuthology.orchestra.run.trial181.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 120/160 2026-02-26T03:23:38.530 INFO:teuthology.orchestra.run.trial181.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 121/160 2026-02-26T03:23:38.547 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 121/160 2026-02-26T03:23:38.661 INFO:teuthology.orchestra.run.trial181.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 122/160 2026-02-26T03:23:38.663 INFO:teuthology.orchestra.run.trial181.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 123/160 2026-02-26T03:23:38.931 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 123/160 2026-02-26T03:23:38.931 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T03:23:38.932 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:39.190 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 124/160 2026-02-26T03:23:39.232 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 124/160 2026-02-26T03:23:39.232 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:23:39.232 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:23:39.233 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:39.236 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 125/160 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 125/160 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout:skipping the directory /sys 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout:skipping the directory /proc 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout:skipping the directory /mnt 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout:skipping the directory /var/tmp 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout:skipping the directory /home 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout:skipping the directory /root 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout:skipping the directory /tmp 2026-02-26T03:23:43.894 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:43.982 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 126/160 2026-02-26T03:23:44.005 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 126/160 2026-02-26T03:23:44.005 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:23:44.005 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:23:44.005 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:23:44.005 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:23:44.005 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:44.215 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 127/160 2026-02-26T03:23:44.236 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 127/160 2026-02-26T03:23:44.236 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:23:44.237 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:23:44.237 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:23:44.237 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:23:44.237 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:44.242 INFO:teuthology.orchestra.run.trial181.stdout: Installing : mailcap-2.1.49-5.el9.noarch 128/160 2026-02-26T03:23:44.244 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 129/160 2026-02-26T03:23:44.261 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 130/160 2026-02-26T03:23:44.261 INFO:teuthology.orchestra.run.trial181.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:23:44.261 INFO:teuthology.orchestra.run.trial181.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:23:44.261 INFO:teuthology.orchestra.run.trial181.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:23:44.261 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:44.270 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 130/160 2026-02-26T03:23:44.293 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 130/160 2026-02-26T03:23:44.293 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:23:44.293 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:44.329 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 131/160 2026-02-26T03:23:44.923 INFO:teuthology.orchestra.run.trial181.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 132/160 2026-02-26T03:23:50.321 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86 133/160 2026-02-26T03:23:50.383 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86 134/160 2026-02-26T03:23:50.385 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 135/160 2026-02-26T03:23:50.398 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 135/160 2026-02-26T03:23:50.399 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:23:50.399 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:23:50.399 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:50.409 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 136/160 2026-02-26T03:23:50.432 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 136/160 2026-02-26T03:23:50.432 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:23:50.432 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:23:50.432 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:23:50.433 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:23:50.433 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:50.492 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 137/160 2026-02-26T03:23:50.494 INFO:teuthology.orchestra.run.trial181.stdout: Installing : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 137/160 2026-02-26T03:23:50.503 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.e 138/160 2026-02-26T03:23:50.527 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617. 139/160 2026-02-26T03:23:50.528 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noa 140/160 2026-02-26T03:23:53.440 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noa 140/160 2026-02-26T03:23:53.446 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.n 141/160 2026-02-26T03:23:53.901 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.n 141/160 2026-02-26T03:23:53.902 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabf 142/160 2026-02-26T03:23:53.913 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabf 142/160 2026-02-26T03:23:53.914 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.n 143/160 2026-02-26T03:23:53.975 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.n 143/160 2026-02-26T03:23:54.016 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el 144/160 2026-02-26T03:23:54.017 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 145/160 2026-02-26T03:23:54.037 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 145/160 2026-02-26T03:23:54.037 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:23:54.037 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:23:54.037 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:23:54.037 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:23:54.037 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:23:54.048 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 146/160 2026-02-26T03:23:56.297 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 146/160 2026-02-26T03:24:18.716 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86 147/160 2026-02-26T03:24:18.717 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb6 148/160 2026-02-26T03:24:19.206 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 149/160 2026-02-26T03:24:19.208 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 150/160 2026-02-26T03:24:19.230 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 150/160 2026-02-26T03:24:19.230 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:19.230 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:24:19.230 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:24:19.230 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:24:19.230 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:19.239 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.gabffb 151/160 2026-02-26T03:24:19.258 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.gabffb 151/160 2026-02-26T03:24:19.258 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:19.258 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:24:19.258 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:19.377 INFO:teuthology.orchestra.run.trial181.stdout: Installing : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 152/160 2026-02-26T03:24:19.396 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 152/160 2026-02-26T03:24:19.396 INFO:teuthology.orchestra.run.trial181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:24:19.396 INFO:teuthology.orchestra.run.trial181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:24:19.396 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:24:19.397 INFO:teuthology.orchestra.run.trial181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:24:19.397 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:26.421 INFO:teuthology.orchestra.run.trial181.stdout: Installing : python3-pytest-6.2.2-7.el9.noarch 153/160 2026-02-26T03:24:26.430 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 154/160 2026-02-26T03:24:26.434 INFO:teuthology.orchestra.run.trial181.stdout: Installing : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_ 155/160 2026-02-26T03:24:26.443 INFO:teuthology.orchestra.run.trial181.stdout: Installing : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 156/160 2026-02-26T03:24:26.485 INFO:teuthology.orchestra.run.trial181.stdout: Installing : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 157/160 2026-02-26T03:24:26.488 INFO:teuthology.orchestra.run.trial181.stdout: Installing : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 158/160 2026-02-26T03:24:26.488 INFO:teuthology.orchestra.run.trial181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/160 2026-02-26T03:24:26.500 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/160 2026-02-26T03:24:26.500 INFO:teuthology.orchestra.run.trial181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:26.511 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:26.517 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86 160/160 2026-02-26T03:24:26.542 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86 160/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 1/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 3/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86 4/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 5/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.gabffb 6/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 7/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 8/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 9/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 10/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86 11/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86 12/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb6 13/160 2026-02-26T03:24:29.539 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 14/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 15/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 16/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86 17/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_ 18/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86 19/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 20/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 21/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_6 22/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86 23/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 24/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el 25/160 2026-02-26T03:24:29.540 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.gabffb617.el9. 26/160 2026-02-26T03:24:29.541 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_6 27/160 2026-02-26T03:24:29.541 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 28/160 2026-02-26T03:24:29.541 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 29/160 2026-02-26T03:24:29.541 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 30/160 2026-02-26T03:24:29.541 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 31/160 2026-02-26T03:24:29.541 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 32/160 2026-02-26T03:24:29.541 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 33/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617. 34/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noa 35/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.n 36/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabf 37/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.n 38/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el 39/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 40/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.e 41/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 42/160 2026-02-26T03:24:29.552 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 43/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 44/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 45/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 46/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 47/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 48/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 49/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 50/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 51/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 52/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 53/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 54/160 2026-02-26T03:24:29.553 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 55/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 56/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 57/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 58/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 59/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 60/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 61/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : unzip-6.0-59.el9.x86_64 62/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : zip-3.0-35.el9.x86_64 63/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 64/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 65/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 66/160 2026-02-26T03:24:29.554 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 67/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 68/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 69/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 73/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 74/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 75/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 76/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 77/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 78/160 2026-02-26T03:24:29.555 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 79/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 80/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 81/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 82/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-iniconfig-1.1.1-7.el9.noarch 83/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 84/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 85/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 86/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 87/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 88/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 89/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 90/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 91/160 2026-02-26T03:24:29.556 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-pluggy-0.13.1-7.el9.noarch 92/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 93/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-py-1.10.0-6.el9.noarch 94/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 95/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 96/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-pytest-6.2.2-7.el9.noarch 97/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 98/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 99/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 100/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 101/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 102/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 103/160 2026-02-26T03:24:29.557 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 104/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 105/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 106/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 107/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 108/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 109/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 110/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 111/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 112/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 113/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 114/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 115/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 116/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 117/160 2026-02-26T03:24:29.558 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 118/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 119/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 120/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 121/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 122/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 123/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 124/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 125/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 126/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 127/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 128/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 129/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 130/160 2026-02-26T03:24:29.559 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 131/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 132/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 133/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 134/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 135/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 136/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 137/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 138/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 139/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 140/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 141/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 142/160 2026-02-26T03:24:29.560 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 143/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 144/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 145/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 146/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 147/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 148/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 149/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 150/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 151/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 152/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 153/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 154/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 155/160 2026-02-26T03:24:29.561 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 156/160 2026-02-26T03:24:29.562 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 157/160 2026-02-26T03:24:29.562 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 158/160 2026-02-26T03:24:29.562 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 159/160 2026-02-26T03:24:29.623 INFO:teuthology.orchestra.run.trial181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 160/160 2026-02-26T03:24:29.623 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:29.623 INFO:teuthology.orchestra.run.trial181.stdout:Upgraded: 2026-02-26T03:24:29.623 INFO:teuthology.orchestra.run.trial181.stdout: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.623 INFO:teuthology.orchestra.run.trial181.stdout: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.623 INFO:teuthology.orchestra.run.trial181.stdout:Installed: 2026-02-26T03:24:29.623 INFO:teuthology.orchestra.run.trial181.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.624 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.625 INFO:teuthology.orchestra.run.trial181.stdout: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-26T03:24:29.626 INFO:teuthology.orchestra.run.trial181.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:24:29.627 INFO:teuthology.orchestra.run.trial181.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:24:29.628 INFO:teuthology.orchestra.run.trial181.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:24:29.629 INFO:teuthology.orchestra.run.trial181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:24:29.630 INFO:teuthology.orchestra.run.trial181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-iniconfig-1.1.1-7.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:24:29.631 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:24:29.632 INFO:teuthology.orchestra.run.trial181.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-pluggy-0.13.1-7.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-py-1.10.0-6.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:24:29.633 INFO:teuthology.orchestra.run.trial181.stdout: python3-pytest-6.2.2-7.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:24:29.634 INFO:teuthology.orchestra.run.trial181.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:24:29.635 INFO:teuthology.orchestra.run.trial181.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:29.636 INFO:teuthology.orchestra.run.trial181.stdout:Complete! 2026-02-26T03:24:29.776 DEBUG:teuthology.parallel:result is None 2026-02-26T03:24:29.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:24:29.944 DEBUG:teuthology.orchestra.run.trial181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:24:29.966 INFO:teuthology.orchestra.run.trial181.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:24:29.967 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:24:29.967 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:24:29.969 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:24:29.969 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:29.969 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:24:30.034 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:24:30.034 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:30.034 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:24:30.097 DEBUG:teuthology.orchestra.run.trial181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:24:30.162 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:24:30.163 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:30.163 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:24:30.226 DEBUG:teuthology.orchestra.run.trial181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:24:30.291 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:24:30.291 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:30.291 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:24:30.354 DEBUG:teuthology.orchestra.run.trial181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:24:30.419 INFO:teuthology.run_tasks:Running task pexec... 2026-02-26T03:24:30.437 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-26T03:24:30.438 DEBUG:teuthology.orchestra.run.trial181:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-26T03:24:30.462 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:24:30.462 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-26T03:24:30.513 DEBUG:teuthology.parallel:result is None 2026-02-26T03:24:30.514 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:24:30.525 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:24:30.525 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:24:30.525 DEBUG:teuthology.task.install:config {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:24:30.525 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:24:30.528 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:24:30.528 INFO:teuthology.task.install:extra packages: [] 2026-02-26T03:24:30.528 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['python3-pytest'], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:24:30.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:24:30.644 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:24:30.644 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:24:31.006 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:24:31.007 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:31.007 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:24:31.032 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-pytest on remote rpm x86_64 2026-02-26T03:24:31.032 DEBUG:teuthology.orchestra.run.trial181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:24:31.096 DEBUG:teuthology.orchestra.run.trial181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:24:31.161 DEBUG:teuthology.orchestra.run.trial181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:24:31.215 INFO:teuthology.orchestra.run.trial181.stdout:check_obsoletes = 1 2026-02-26T03:24:31.225 DEBUG:teuthology.orchestra.run.trial181:> sudo yum clean all 2026-02-26T03:24:31.386 INFO:teuthology.orchestra.run.trial181.stdout:56 files removed 2026-02-26T03:24:31.401 DEBUG:teuthology.orchestra.run.trial181:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-osd ceph-osd-classic ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-pytest 2026-02-26T03:24:32.359 INFO:teuthology.orchestra.run.trial181.stdout:ceph packages for x86_64 117 kB/s | 94 kB 00:00 2026-02-26T03:24:33.099 INFO:teuthology.orchestra.run.trial181.stdout:ceph noarch packages 34 kB/s | 24 kB 00:00 2026-02-26T03:24:33.701 INFO:teuthology.orchestra.run.trial181.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:24:34.652 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.9 MB 00:00 2026-02-26T03:24:37.744 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-26T03:24:41.288 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 8.0 MB 00:01 2026-02-26T03:24:42.839 INFO:teuthology.orchestra.run.trial181.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 20 kB 00:00 2026-02-26T03:24:44.176 INFO:teuthology.orchestra.run.trial181.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2026-02-26T03:24:47.473 INFO:teuthology.orchestra.run.trial181.stdout:lab-extras 1.4 MB/s | 50 kB 00:00 2026-02-26T03:24:48.672 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.673 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.673 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.673 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.674 INFO:teuthology.orchestra.run.trial181.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:24:48.674 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.674 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.675 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:24:48.675 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:24:48.676 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:24:48.676 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:24:48.677 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.677 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.677 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.678 INFO:teuthology.orchestra.run.trial181.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:24:48.678 INFO:teuthology.orchestra.run.trial181.stdout:Package librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.679 INFO:teuthology.orchestra.run.trial181.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.679 INFO:teuthology.orchestra.run.trial181.stdout:Package libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.680 INFO:teuthology.orchestra.run.trial181.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.680 INFO:teuthology.orchestra.run.trial181.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.681 INFO:teuthology.orchestra.run.trial181.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.681 INFO:teuthology.orchestra.run.trial181.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.681 INFO:teuthology.orchestra.run.trial181.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.682 INFO:teuthology.orchestra.run.trial181.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.682 INFO:teuthology.orchestra.run.trial181.stdout:Package rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.683 INFO:teuthology.orchestra.run.trial181.stdout:Package rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.683 INFO:teuthology.orchestra.run.trial181.stdout:Package rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:24:48.683 INFO:teuthology.orchestra.run.trial181.stdout:Package python3-pytest-6.2.2-7.el9.noarch is already installed. 2026-02-26T03:24:48.702 INFO:teuthology.orchestra.run.trial181.stdout:Dependencies resolved. 2026-02-26T03:24:48.702 INFO:teuthology.orchestra.run.trial181.stdout:Nothing to do. 2026-02-26T03:24:48.703 INFO:teuthology.orchestra.run.trial181.stdout:Complete! 2026-02-26T03:24:48.731 DEBUG:teuthology.parallel:result is None 2026-02-26T03:24:48.732 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:24:48.895 DEBUG:teuthology.orchestra.run.trial181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:24:48.916 INFO:teuthology.orchestra.run.trial181.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:24:48.916 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:24:48.916 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:24:48.918 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-26T03:24:48.918 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:24:49.082 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:24:49.082 DEBUG:teuthology.orchestra.run.trial181:> sudo chmod 777 /var/log/ceph 2026-02-26T03:24:49.108 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:24:49.108 DEBUG:teuthology.orchestra.run.trial181:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:24:49.172 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:24:49.172 DEBUG:teuthology.orchestra.run.trial181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:24:49.245 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:24:49.245 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:24:49.245 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71183', 'branch': 'main', 'description': 'crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} tasks/rbd_python_api_tests_old_format}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '71183', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5117, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996', 'targets': {'trial181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNUXvXM4NpDN/m5Krs+I785215/eVMa2OjDs4Qu7FLCkWwq77aDcpHVLu1BF+Yn+do/wzzrm4LRAFSS9wuffLss='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': ['python3-pytest'], 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/test_librbd_python.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_02:22:56', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:24:49.245 DEBUG:teuthology.orchestra.run.trial181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:24:49.294 DEBUG:teuthology.orchestra.run.trial181:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:24:49.360 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:49.361 DEBUG:teuthology.orchestra.run.trial181:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:24:49.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:24:49.418 DEBUG:teuthology.orchestra.run.trial181:> stat /dev/vg_nvme/lv_1 2026-02-26T03:24:49.474 INFO:teuthology.orchestra.run.trial181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:24:49.474 INFO:teuthology.orchestra.run.trial181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:24:49.474 INFO:teuthology.orchestra.run.trial181.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-26T03:24:49.474 INFO:teuthology.orchestra.run.trial181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:24:49.475 INFO:teuthology.orchestra.run.trial181.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:24:49.475 INFO:teuthology.orchestra.run.trial181.stdout:Access: 2026-02-26 03:24:26.742377049 +0000 2026-02-26T03:24:49.475 INFO:teuthology.orchestra.run.trial181.stdout:Modify: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.475 INFO:teuthology.orchestra.run.trial181.stdout:Change: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.475 INFO:teuthology.orchestra.run.trial181.stdout: Birth: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.475 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:24:49.537 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records in 2026-02-26T03:24:49.538 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records out 2026-02-26T03:24:49.538 INFO:teuthology.orchestra.run.trial181.stderr:512 bytes copied, 0.000169445 s, 3.0 MB/s 2026-02-26T03:24:49.539 DEBUG:teuthology.orchestra.run.trial181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:24:49.596 DEBUG:teuthology.orchestra.run.trial181:> stat /dev/vg_nvme/lv_2 2026-02-26T03:24:49.652 INFO:teuthology.orchestra.run.trial181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:24:49.652 INFO:teuthology.orchestra.run.trial181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:24:49.653 INFO:teuthology.orchestra.run.trial181.stdout:Device: 6h/6d Inode: 1303 Links: 1 2026-02-26T03:24:49.653 INFO:teuthology.orchestra.run.trial181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:24:49.653 INFO:teuthology.orchestra.run.trial181.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:24:49.653 INFO:teuthology.orchestra.run.trial181.stdout:Access: 2026-02-26 03:24:26.742377049 +0000 2026-02-26T03:24:49.653 INFO:teuthology.orchestra.run.trial181.stdout:Modify: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.653 INFO:teuthology.orchestra.run.trial181.stdout:Change: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.653 INFO:teuthology.orchestra.run.trial181.stdout: Birth: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.653 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:24:49.715 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records in 2026-02-26T03:24:49.715 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records out 2026-02-26T03:24:49.715 INFO:teuthology.orchestra.run.trial181.stderr:512 bytes copied, 0.000142224 s, 3.6 MB/s 2026-02-26T03:24:49.717 DEBUG:teuthology.orchestra.run.trial181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:24:49.774 DEBUG:teuthology.orchestra.run.trial181:> stat /dev/vg_nvme/lv_3 2026-02-26T03:24:49.830 INFO:teuthology.orchestra.run.trial181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:24:49.830 INFO:teuthology.orchestra.run.trial181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:24:49.830 INFO:teuthology.orchestra.run.trial181.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-26T03:24:49.830 INFO:teuthology.orchestra.run.trial181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:24:49.830 INFO:teuthology.orchestra.run.trial181.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:24:49.830 INFO:teuthology.orchestra.run.trial181.stdout:Access: 2026-02-26 03:24:26.743377025 +0000 2026-02-26T03:24:49.830 INFO:teuthology.orchestra.run.trial181.stdout:Modify: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.831 INFO:teuthology.orchestra.run.trial181.stdout:Change: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.831 INFO:teuthology.orchestra.run.trial181.stdout: Birth: 2026-02-26 03:23:38.958568269 +0000 2026-02-26T03:24:49.831 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:24:49.893 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records in 2026-02-26T03:24:49.893 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records out 2026-02-26T03:24:49.893 INFO:teuthology.orchestra.run.trial181.stderr:512 bytes copied, 0.000181728 s, 2.8 MB/s 2026-02-26T03:24:49.895 DEBUG:teuthology.orchestra.run.trial181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:24:49.950 DEBUG:teuthology.orchestra.run.trial181:> stat /dev/vg_nvme/lv_4 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout:Device: 6h/6d Inode: 1293 Links: 1 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout:Access: 2026-02-26 03:24:26.743377025 +0000 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout:Modify: 2026-02-26 03:23:38.957568294 +0000 2026-02-26T03:24:50.007 INFO:teuthology.orchestra.run.trial181.stdout:Change: 2026-02-26 03:23:38.957568294 +0000 2026-02-26T03:24:50.008 INFO:teuthology.orchestra.run.trial181.stdout: Birth: 2026-02-26 03:23:38.957568294 +0000 2026-02-26T03:24:50.008 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:24:50.070 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records in 2026-02-26T03:24:50.070 INFO:teuthology.orchestra.run.trial181.stderr:1+0 records out 2026-02-26T03:24:50.070 INFO:teuthology.orchestra.run.trial181.stderr:512 bytes copied, 0.000143556 s, 3.6 MB/s 2026-02-26T03:24:50.072 DEBUG:teuthology.orchestra.run.trial181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:24:50.128 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:24:50.129 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:24:50.129 INFO:tasks.ceph:Generating config... 2026-02-26T03:24:50.131 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-26T03:24:50.132 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-26T03:24:50.132 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-26T03:24:50.133 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-26T03:24:50.134 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-26T03:24:50.134 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:24:50.134 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-26T03:24:50.134 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-26T03:24:50.135 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-26T03:24:50.135 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-26T03:24:50.135 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-26T03:24:50.135 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-26T03:24:50.135 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:24:50.135 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:24:50.135 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:24:50.135 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:24:50.208 INFO:teuthology.orchestra.run.trial181.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:24:50.211 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:24:50.295 DEBUG:teuthology.orchestra.run.trial181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:24:50.359 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.181')] 2026-02-26T03:24:50.360 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.181', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:24:50.361 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:50.361 DEBUG:teuthology.orchestra.run.trial181:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:24:50.415 DEBUG:teuthology.orchestra.run.trial181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.181 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:24:50.495 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:24:50.493+0000 7fc39ccc8f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:24:50.496 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:24:50.494+0000 7fc39ccc8f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:24:50.496 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:24:50.494+0000 7fc39ccc8f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:24:50.497 INFO:teuthology.orchestra.run.trial181.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:24:50.497 INFO:teuthology.orchestra.run.trial181.stdout:monmaptool: generated fsid 7aaab781-9a96-4919-9f35-646a2d4e5c22 2026-02-26T03:24:50.497 INFO:teuthology.orchestra.run.trial181.stdout:setting min_mon_release = tentacle 2026-02-26T03:24:50.497 INFO:teuthology.orchestra.run.trial181.stdout:epoch 0 2026-02-26T03:24:50.497 INFO:teuthology.orchestra.run.trial181.stdout:fsid 7aaab781-9a96-4919-9f35-646a2d4e5c22 2026-02-26T03:24:50.498 INFO:teuthology.orchestra.run.trial181.stdout:last_changed 2026-02-26T03:24:50.495344+0000 2026-02-26T03:24:50.498 INFO:teuthology.orchestra.run.trial181.stdout:created 2026-02-26T03:24:50.495344+0000 2026-02-26T03:24:50.498 INFO:teuthology.orchestra.run.trial181.stdout:min_mon_release 20 (tentacle) 2026-02-26T03:24:50.498 INFO:teuthology.orchestra.run.trial181.stdout:election_strategy: 1 2026-02-26T03:24:50.498 INFO:teuthology.orchestra.run.trial181.stdout:0: [v2:10.20.193.181:3300/0,v1:10.20.193.181:6789/0] mon.a 2026-02-26T03:24:50.498 INFO:teuthology.orchestra.run.trial181.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:24:50.499 DEBUG:teuthology.orchestra.run.trial181:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:24:50.554 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7aaab781-9a96-4919-9f35-646a2d4e5c22... 2026-02-26T03:24:50.554 DEBUG:teuthology.orchestra.run.trial181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:24:50.633 INFO:teuthology.orchestra.run.trial181.stdout:[global] 2026-02-26T03:24:50.633 INFO:teuthology.orchestra.run.trial181.stdout: chdir = "" 2026-02-26T03:24:50.633 INFO:teuthology.orchestra.run.trial181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: auth supported = cephx 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: filestore xattr use omap = true 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: mon clock drift allowed = 1.000 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: osd crush chooseleaf type = 0 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: auth debug = true 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: ms die on old message = true 2026-02-26T03:24:50.634 INFO:teuthology.orchestra.run.trial181.stdout: ms die on bug = true 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: mon pg warn max object skew = 0 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: osd pool default size = 2 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: mon osd allow primary affinity = true 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: mon osd allow pg remap = true 2026-02-26T03:24:50.635 INFO:teuthology.orchestra.run.trial181.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon warn on no sortbitwise = false 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon warn on too few osds = false 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon_allow_pool_size_one = true 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: osd default data pool replay window = 5 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.636 INFO:teuthology.orchestra.run.trial181.stdout: mon allow pool delete = true 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: mon cluster log file level = debug 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: debug asserts on shutdown = true 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: mon health detail to clog = false 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: mon host = 10.20.193.181 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: ms client mode = crc 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: ms cluster mode = crc 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: ms mon client mode = crc 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: ms mon cluster mode = crc 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: ms mon service mode = crc 2026-02-26T03:24:50.637 INFO:teuthology.orchestra.run.trial181.stdout: ms service mode = crc 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: fsid = 7aaab781-9a96-4919-9f35-646a2d4e5c22 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout:[osd] 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: osd journal size = 100 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: osd scrub load threshold = 5.0 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: osd scrub max interval = 600 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: osd mclock skip benchmark = true 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: osd recover clone overlap = true 2026-02-26T03:24:50.638 INFO:teuthology.orchestra.run.trial181.stdout: osd recovery max chunk = 1048576 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd debug shutdown = true 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd debug op order = true 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd debug verify stray on activate = true 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd debug trim objects = true 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd open classes on start = true 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd debug pg log writeout = true 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: osd map max advance = 10 2026-02-26T03:24:50.639 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: journal zero on create = true 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: filestore apply finisher threads = 3 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: bdev debug aio = true 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: osd debug misdirected ops = true 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: debug monc = 20 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: crimson cpu num = 4 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: crimson osd obc lru size = 10 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: debug ms = 20 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: debug osd = 20 2026-02-26T03:24:50.640 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore backref = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore cache = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore cleaner = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore device = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore epm = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore fixedkv tree = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore journal = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore lba = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore odata = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore omap = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore onode = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore t = 20 2026-02-26T03:24:50.641 INFO:teuthology.orchestra.run.trial181.stdout: debug seastore tm = 20 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: osd objectstore = seastore 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: osd shutdown pgref assert = True 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: seastore cachepin size pershard = 64M 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: seastore max concurrent transactions = 8 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout:[mgr] 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: debug ms = 1 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: debug mgr = 20 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: debug mon = 20 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: debug auth = 20 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:24:50.642 INFO:teuthology.orchestra.run.trial181.stdout: mgr/telemetry/nag = false 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout:[mon] 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: debug ms = 1 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: debug mon = 20 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: debug paxos = 20 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: debug auth = 20 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: mon data avail warn = 5 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: mon mgr mkfs grace = 240 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: mon osd reporter subtree level = osd 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: mon osd prime pg temp = true 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:24:50.643 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: # 1m isn't quite enough 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:24:50.644 INFO:teuthology.orchestra.run.trial181.stdout: mon min osdmap epochs = 50 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: mon osdmap full prune interval = 2 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: mon osdmap full prune min = 15 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: mon osdmap full prune txsize = 2 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: paxos service trim min = 10 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: osd pool default crimson = True 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout:[client] 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: rgw cache enabled = true 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: rgw enable ops log = true 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: rgw enable usage log = true 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:24:50.645 INFO:teuthology.orchestra.run.trial181.stdout:[mon.a] 2026-02-26T03:24:50.647 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:24:50.647 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:24:50.737 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:24:50.737 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:50.737 DEBUG:teuthology.orchestra.run.trial181:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:24:50.754 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:50.754 DEBUG:teuthology.orchestra.run.trial181:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:24:50.810 INFO:tasks.ceph:Sending monmap to node ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:24:50.810 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:50.810 DEBUG:teuthology.orchestra.run.trial181:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:24:50.810 DEBUG:teuthology.orchestra.run.trial181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:24:50.883 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:24:50.883 DEBUG:teuthology.orchestra.run.trial181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:24:50.940 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:24:50.940 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:24:50.940 DEBUG:teuthology.orchestra.run.trial181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:24:51.029 INFO:teuthology.orchestra.run.trial181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:24:51.032 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:24:51.032 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:24:51.033 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:24:51.074 INFO:teuthology.orchestra.run.trial181.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:24:51.086 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:24:51.087 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:24:51.087 DEBUG:teuthology.orchestra.run.trial181:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:24:51.150 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:24:51.150 INFO:tasks.ceph:role: osd.0 2026-02-26T03:24:51.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:24:51.151 DEBUG:teuthology.orchestra.run.trial181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:24:51.215 INFO:teuthology.orchestra.run.trial181.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:24:51.216 INFO:teuthology.orchestra.run.trial181.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:24:51.216 INFO:teuthology.orchestra.run.trial181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:24:51.785 INFO:teuthology.orchestra.run.trial181.stdout:Discarding blocks...Done. 2026-02-26T03:24:51.787 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial181.front.sepia.ceph.com -o noatime 2026-02-26T03:24:51.788 DEBUG:teuthology.orchestra.run.trial181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:24:51.856 DEBUG:teuthology.orchestra.run.trial181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:24:51.925 DEBUG:teuthology.orchestra.run.trial181:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:24:51.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:24:51.989 INFO:tasks.ceph:role: osd.1 2026-02-26T03:24:51.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:24:51.989 DEBUG:teuthology.orchestra.run.trial181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:24:52.055 INFO:teuthology.orchestra.run.trial181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:24:52.055 INFO:teuthology.orchestra.run.trial181.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:24:52.055 INFO:teuthology.orchestra.run.trial181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:24:52.055 INFO:teuthology.orchestra.run.trial181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:24:52.055 INFO:teuthology.orchestra.run.trial181.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:24:52.056 INFO:teuthology.orchestra.run.trial181.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:24:52.056 INFO:teuthology.orchestra.run.trial181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:24:52.056 INFO:teuthology.orchestra.run.trial181.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:24:52.056 INFO:teuthology.orchestra.run.trial181.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:24:52.056 INFO:teuthology.orchestra.run.trial181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:24:52.613 INFO:teuthology.orchestra.run.trial181.stdout:Discarding blocks...Done. 2026-02-26T03:24:52.615 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial181.front.sepia.ceph.com -o noatime 2026-02-26T03:24:52.615 DEBUG:teuthology.orchestra.run.trial181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:24:52.684 DEBUG:teuthology.orchestra.run.trial181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:24:52.751 DEBUG:teuthology.orchestra.run.trial181:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:24:52.814 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:24:52.814 INFO:tasks.ceph:role: osd.2 2026-02-26T03:24:52.814 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:24:52.815 DEBUG:teuthology.orchestra.run.trial181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:24:52.879 INFO:teuthology.orchestra.run.trial181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:24:52.879 INFO:teuthology.orchestra.run.trial181.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:24:52.879 INFO:teuthology.orchestra.run.trial181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:24:52.879 INFO:teuthology.orchestra.run.trial181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:24:52.879 INFO:teuthology.orchestra.run.trial181.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:24:52.880 INFO:teuthology.orchestra.run.trial181.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:24:52.880 INFO:teuthology.orchestra.run.trial181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:24:52.880 INFO:teuthology.orchestra.run.trial181.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:24:52.880 INFO:teuthology.orchestra.run.trial181.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:24:52.880 INFO:teuthology.orchestra.run.trial181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:24:53.438 INFO:teuthology.orchestra.run.trial181.stdout:Discarding blocks...Done. 2026-02-26T03:24:53.440 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial181.front.sepia.ceph.com -o noatime 2026-02-26T03:24:53.441 DEBUG:teuthology.orchestra.run.trial181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:24:53.507 DEBUG:teuthology.orchestra.run.trial181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:24:53.574 DEBUG:teuthology.orchestra.run.trial181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:24:53.725 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:24:53.777 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:53,776 seastar - Reactor backend: linux-aio 2026-02-26T03:24:53.788 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:53,787 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:24:53.788 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:53,787 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:24:53.790 INFO:teuthology.orchestra.run.trial181.stderr:==37055==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:24:53.844 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:53,844 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:24:53.845 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:53,844 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:24:53.845 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:53,844 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:24:53.845 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:53,844 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:24:53.845 INFO:teuthology.orchestra.run.trial181.stderr:DEBUG 2026-02-26 03:24:53,844 [shard 0:main] seastar - reactor::drain 2026-02-26T03:24:55.873 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:24:55.874 INFO:teuthology.orchestra.run.trial181.stderr:Suppressions used: 2026-02-26T03:24:55.874 INFO:teuthology.orchestra.run.trial181.stderr: count bytes template 2026-02-26T03:24:55.874 INFO:teuthology.orchestra.run.trial181.stderr: 1 8 MallocExtension::Register 2026-02-26T03:24:55.874 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:24:55.874 INFO:teuthology.orchestra.run.trial181.stderr: 2026-02-26T03:24:55.892 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:24:55.893 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:24:55.934 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:24:55.934 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:24:55.941 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:55,940 seastar - Reactor backend: linux-aio 2026-02-26T03:24:55.952 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:55,952 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:24:55.952 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:55,952 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:24:55.955 INFO:teuthology.orchestra.run.trial181.stderr:==37053==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:24:55.955 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:55,955 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:24:56.014 INFO:teuthology.orchestra.run.trial181.stderr:DEBUG 2026-02-26 03:24:56,014 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:24:56.093 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x157b4 0x10908ef 0xf2e0dc 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d320be 0x9d32837 0x9d32d22 0x9d330b5 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.094 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libstdc++.so.6+0xc6227 0x316e07f 0x9c50ec9 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.094 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x109078b 0xf2e0dc 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c52ef3 0x9c53440 0x9c5363f 0x9d38961 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.126 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x16507 0x7b974c2 0x7becb21 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa14d6 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:24:56.163 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54d33 0x7d837e7 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:24:56.167 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x780db94 0x780dc80 0x9b25276 0x9d46c67 0x9d07d91 0x9d06d91 0x7b9b750 0x7bede81 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa14d6 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.231 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54a0c 0x7d837d2 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.232 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15277 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.236 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.366 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xe9369 0xb69825 0xb75aef 0xbb92ea 0xe05f80 0x7c933ed 0x7bdaa69 0x7be9ecd 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.366 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d9 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d38e4d 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.366 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c6313d 0x9c6325f 0x9d23007 0x9d2319c 0x9d2352e 0x9d24b15 0x9d25e30 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.448 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a5490e 0x7d837d2 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:24:56.463 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12978a /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.463 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a5530d 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:56.463 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a559ce 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:24:58.634 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:24:58.634 INFO:teuthology.orchestra.run.trial181.stderr:Suppressions used: 2026-02-26T03:24:58.634 INFO:teuthology.orchestra.run.trial181.stderr: count bytes template 2026-02-26T03:24:58.634 INFO:teuthology.orchestra.run.trial181.stderr: 1 8 MallocExtension::Register 2026-02-26T03:24:58.634 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:24:58.634 INFO:teuthology.orchestra.run.trial181.stderr: 2026-02-26T03:24:58.670 DEBUG:teuthology.orchestra.run.trial181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:24:58.695 DEBUG:teuthology.orchestra.run.trial181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:24:58.846 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:24:58.903 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:58,902 seastar - Reactor backend: linux-aio 2026-02-26T03:24:58.912 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:58,912 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:24:58.912 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:58,912 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:24:58.914 INFO:teuthology.orchestra.run.trial181.stderr:==37121==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:24:58.970 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:58,970 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:24:58.970 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:58,970 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:24:58.970 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:58,970 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:24:58.970 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:24:58,970 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:24:58.970 INFO:teuthology.orchestra.run.trial181.stderr:DEBUG 2026-02-26 03:24:58,970 [shard 0:main] seastar - reactor::drain 2026-02-26T03:25:01.076 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:01.077 INFO:teuthology.orchestra.run.trial181.stderr:Suppressions used: 2026-02-26T03:25:01.077 INFO:teuthology.orchestra.run.trial181.stderr: count bytes template 2026-02-26T03:25:01.077 INFO:teuthology.orchestra.run.trial181.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:01.077 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:01.077 INFO:teuthology.orchestra.run.trial181.stderr: 2026-02-26T03:25:01.097 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:25:01.097 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:25:01.145 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:25:01.145 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:25:01.152 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:01,151 seastar - Reactor backend: linux-aio 2026-02-26T03:25:01.163 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:01,163 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:01.163 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:01,163 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:01.167 INFO:teuthology.orchestra.run.trial181.stderr:==37119==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:01.167 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:01,166 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:25:01.225 INFO:teuthology.orchestra.run.trial181.stderr:DEBUG 2026-02-26 03:25:01,225 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:25:01.305 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15822 0x10908ef 0xf2e0dc 0x9c554b6 0x9c55763 0x9d42c55 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.306 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x157b3 0x10908ef 0xf2e0dc 0x9c554b6 0x9c55df5 0x9d41dc5 0x9d42ce5 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.306 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.338 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x8f5cd78 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:25:01.379 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec3 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x157c7 0xc95eb2 0xca741d 0x9c98955 0x9d47a94 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa14d6 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.380 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb138a 0xa5d7026 0x9d4877c 0x9d48963 0x9d48b2c 0x9d48bbf 0x9d49a04 0x78947dd 0x7b9b15c 0x7bee1a9 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa14d6 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.380 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9ae26a6 0x9bc9b90 0x9bc9beb 0x9bc9d4e 0x9bcaca0 0x9bc7c4b 0x772d8ea 0x77e0616 0x77e2a38 0x943658e 0x947b43c 0x7dd81cb 0x7ee070c 0x7f55cd5 0x7d8b2fd 0x7e6482d 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.404 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54d91 0x7d837e7 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:25:01.417 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a549cb 0x7d837d2 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.445 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55a4b 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.447 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a5540f 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.575 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x748c80d 0x73d690a 0x7159a3e 0x7be997a 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.575 INFO:teuthology.orchestra.run.trial181.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb35d4ac0 0xffffffffb35d500b 0xffffffffb32910d2 0xffffffffb3f50451 0xffffffffb4000ba2 2026-02-26T03:25:01.575 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe1d59b 0x9d48f07 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.575 INFO:teuthology.orchestra.run.trial181.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb341ff44 0xffffffffb34203b0 0xffffffffb329525e 0xffffffffb35e7269 0xffffffffb35ca7d5 0xffffffffb35d4bab 0xffffffffb35d500b 0xffffffffb32910d2 0xffffffffb3f50451 0xffffffffb4000ba2 2026-02-26T03:25:01.575 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libstdc++.so.6+0xc621f 0x316e07f 0x9c50ec9 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.657 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55954 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:25:01.671 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a5599f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.672 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bea 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:01.673 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a5590c 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:03.862 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:03.863 INFO:teuthology.orchestra.run.trial181.stderr:Suppressions used: 2026-02-26T03:25:03.863 INFO:teuthology.orchestra.run.trial181.stderr: count bytes template 2026-02-26T03:25:03.863 INFO:teuthology.orchestra.run.trial181.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:03.863 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:03.863 INFO:teuthology.orchestra.run.trial181.stderr: 2026-02-26T03:25:03.897 DEBUG:teuthology.orchestra.run.trial181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:25:03.921 DEBUG:teuthology.orchestra.run.trial181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:04.072 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:25:04.129 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:04,129 seastar - Reactor backend: linux-aio 2026-02-26T03:25:04.147 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:04,146 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:04.147 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:04,146 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:04.149 INFO:teuthology.orchestra.run.trial181.stderr:==37186==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:04.204 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:04,203 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-26T03:25:04.204 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:04,203 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-26T03:25:04.204 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:04,204 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-26T03:25:04.204 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:04,204 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-26T03:25:04.204 INFO:teuthology.orchestra.run.trial181.stderr:DEBUG 2026-02-26 03:25:04,204 [shard 0:main] seastar - reactor::drain 2026-02-26T03:25:06.311 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:06.311 INFO:teuthology.orchestra.run.trial181.stderr:Suppressions used: 2026-02-26T03:25:06.311 INFO:teuthology.orchestra.run.trial181.stderr: count bytes template 2026-02-26T03:25:06.311 INFO:teuthology.orchestra.run.trial181.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:06.311 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:06.311 INFO:teuthology.orchestra.run.trial181.stderr: 2026-02-26T03:25:06.331 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:25:06.331 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:25:06.376 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:25:06.377 INFO:teuthology.orchestra.run.trial181.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:25:06.384 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:06,384 seastar - Reactor backend: linux-aio 2026-02-26T03:25:06.394 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:06,394 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:06.394 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:06,394 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:25:06.398 INFO:teuthology.orchestra.run.trial181.stderr:==37184==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:25:06.398 INFO:teuthology.orchestra.run.trial181.stderr:INFO 2026-02-26 03:25:06,398 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:25:06.457 INFO:teuthology.orchestra.run.trial181.stderr:DEBUG 2026-02-26 03:25:06,456 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-26T03:25:06.536 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c52ef3 0x9c53440 0x9c5363f 0x9d38961 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.536 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfdf27 /lib64/libasan.so.8+0xc6d7 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba35e4 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9d45991 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.537 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x10908dc 0xf2e0dc 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d323e2 0x9d32837 0x9d32d22 0x9d330b5 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.569 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xa31f /lib64/libasan.so.8+0xc478 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe0b7b5 0xe0b9eb 0x7e5d9f2 0x7e64490 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:25:06.611 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c5f72e 0x9d25a33 0x9d25bfb 0x9d25d5c 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7e61695 0x7e64490 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.635 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bea 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:25:06.675 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.676 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.676 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bc1 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.800 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xc95d53 0xca72ee 0x9c4c02e 0x9d479ea 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.800 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbf3 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.800 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfdf40 /lib64/libasan.so.8+0xeb6b /lib64/libasan.so.8+0xef905 0x31724fc 0x31726d6 0x317298d 0x3172952 0x3172a82 0x9d45d59 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.850 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55837 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:25:06.897 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.898 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54d5d 0x7d837e7 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:06.898 INFO:teuthology.orchestra.run.trial181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a549cb 0x7d837d2 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:25:09.038 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:09.039 INFO:teuthology.orchestra.run.trial181.stderr:Suppressions used: 2026-02-26T03:25:09.039 INFO:teuthology.orchestra.run.trial181.stderr: count bytes template 2026-02-26T03:25:09.039 INFO:teuthology.orchestra.run.trial181.stderr: 1 8 MallocExtension::Register 2026-02-26T03:25:09.039 INFO:teuthology.orchestra.run.trial181.stderr:----------------------------------------------------- 2026-02-26T03:25:09.039 INFO:teuthology.orchestra.run.trial181.stderr: 2026-02-26T03:25:09.074 DEBUG:teuthology.orchestra.run.trial181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:25:09.100 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:25:09.100 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:09.101 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:25:09.165 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:09.165 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:25:09.230 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:09.230 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:25:09.294 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:09.294 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:25:09.359 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:09.359 DEBUG:teuthology.orchestra.run.trial181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:25:09.414 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:25:09.414 DEBUG:teuthology.orchestra.run.trial181:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:25:09.476 INFO:teuthology.orchestra.run.trial181.stdout:[mgr.x] 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout: key = AQADvZ9pDgfBARAAuwoXx+YPEnbHOVrKNDf5Ag== 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout:[osd.0] 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout: key = AQAIvZ9p2zoUARAAwzwFOrpUhXlTSsHfiZ3AhA== 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout:[osd.1] 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout: key = AQANvZ9pa8uiDRAArcTEgmZmHmXL0wf6Cpu6iA== 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout:[osd.2] 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout: key = AQASvZ9pdZd0GxAAbl4NABgx/qkHLd0Na2OKhQ== 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout:[client.0] 2026-02-26T03:25:09.477 INFO:teuthology.orchestra.run.trial181.stdout: key = AQADvZ9pwPNnBBAA1IjDKxXmIZ2+FXJnybJZFg== 2026-02-26T03:25:09.479 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:25:09.566 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:25:09.615 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:25:09.664 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:25:09.711 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:25:09.762 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:25:09.762 DEBUG:teuthology.orchestra.run.trial181:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:25:09.786 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:25:09.872 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:09.870+0000 7f44332aec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:09.872 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:09.870+0000 7f44332aec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:09.884 DEBUG:teuthology.orchestra.run.trial181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:25:09.909 DEBUG:teuthology.orchestra.run.trial181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:25:09.965 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:25:09.965 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:25:09.965 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:25:10.007 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:25:10.007 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:25:10.007 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:25:10.007 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:25:10.008 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:25:10.008 DEBUG:tasks.ceph:set 0 configs 2026-02-26T03:25:10.008 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph config dump 2026-02-26T03:25:10.069 INFO:tasks.ceph.mon.a.trial181.stderr:2026-02-26T03:25:10.067+0000 7fdbfe0d6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.069 INFO:tasks.ceph.mon.a.trial181.stderr:2026-02-26T03:25:10.067+0000 7fdbfe0d6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.069 INFO:tasks.ceph.mon.a.trial181.stderr:2026-02-26T03:25:10.067+0000 7fdbfe0d6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.071 INFO:tasks.ceph.mgr.x.trial181.stderr:2026-02-26T03:25:10.069+0000 7f63e95bffc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.083 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:10.080+0000 7f1d3aecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.090 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:10.089+0000 7f1d3aecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.123 INFO:teuthology.orchestra.run.trial181.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-26T03:25:10.132 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:25:10.132 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:25:10.205 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:10.201+0000 7f607965a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.208 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:10.206+0000 7f607965a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.242 INFO:teuthology.orchestra.run.trial181.stderr:adjusted tunables profile to default 2026-02-26T03:25:10.250 INFO:tasks.ceph:check_enable_crimson: True 2026-02-26T03:25:10.251 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-26T03:25:10.251 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-26T03:25:10.274 INFO:tasks.ceph.mgr.x.trial181.stderr:2026-02-26T03:25:10.273+0000 7f63e95bffc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.275 INFO:tasks.ceph.mgr.x.trial181.stderr:2026-02-26T03:25:10.273+0000 7f63e95bffc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.324 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:10.320+0000 7f42c5d93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.326 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:10.324+0000 7f42c5d93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:10.365 INFO:tasks.ceph.mon.a.trial181.stderr:2026-02-26T03:25:10.364+0000 7fdbf57f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-26T03:25:10.366 INFO:tasks.ceph.mon.a.trial181.stderr:Please be aware that this feature is experimental, untested, 2026-02-26T03:25:10.366 INFO:tasks.ceph.mon.a.trial181.stderr:unsupported, and may result in data corruption, data loss, 2026-02-26T03:25:10.366 INFO:tasks.ceph.mon.a.trial181.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-26T03:25:10.366 INFO:tasks.ceph.mon.a.trial181.stderr:feature with important data. 2026-02-26T03:25:10.366 INFO:tasks.ceph.mon.a.trial181.stderr: 2026-02-26T03:25:11.123 INFO:tasks.ceph.mgr.x.trial181.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:25:11.124 INFO:tasks.ceph.mgr.x.trial181.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:25:11.124 INFO:tasks.ceph.mgr.x.trial181.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:25:11.252 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:25:11.253 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:11.253 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:25:11.320 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:11.320 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:25:11.383 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:11.383 DEBUG:teuthology.orchestra.run.trial181:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:25:11.448 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph osd new 7965c363-a204-48ea-b372-753f691a6991 0 2026-02-26T03:25:11.562 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:11.558+0000 7f94759ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:11.565 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:11.563+0000 7f94759ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:11.603 INFO:teuthology.orchestra.run.trial181.stdout:0 2026-02-26T03:25:11.611 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph osd new 8435dd82-163b-48e9-9691-65b13227e0f2 1 2026-02-26T03:25:11.684 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:11.680+0000 7fb097de4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:11.687 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:11.686+0000 7fb097de4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:11.726 INFO:teuthology.orchestra.run.trial181.stdout:1 2026-02-26T03:25:11.735 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph osd new 5918fd4e-82ea-4938-9231-f307a16b051c 2 2026-02-26T03:25:11.808 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:11.803+0000 7f707383d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:11.811 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:11.809+0000 7f707383d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:11.850 INFO:teuthology.orchestra.run.trial181.stdout:2 2026-02-26T03:25:11.858 DEBUG:teuthology.orchestra.run.trial181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-26T03:25:11.886 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:25:11.886 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-26T03:25:11.927 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:25:11.928 DEBUG:teuthology.orchestra.run.trial181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-26T03:25:11.956 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:25:11.956 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-26T03:25:11.999 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:25:11.999 DEBUG:teuthology.orchestra.run.trial181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-26T03:25:12.025 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:25:12.026 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-26T03:25:12.069 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:25:12.069 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:12.144 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:12.140+0000 7fbf23e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:12.146 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:12.145+0000 7fbf23e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:12.186 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:12.186 INFO:teuthology.orchestra.run.trial181.stdout:{"epoch":6,"fsid":"7aaab781-9a96-4919-9f35-646a2d4e5c22","created":"2026-02-26T03:25:10.083274+0000","modified":"2026-02-26T03:25:11.849219+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:25:11.849219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7965c363-a204-48ea-b372-753f691a6991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8435dd82-163b-48e9-9691-65b13227e0f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5918fd4e-82ea-4938-9231-f307a16b051c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:12.195 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:25:12.195 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:25:12.497 DEBUG:teuthology.orchestra.run.trial181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:12.565 INFO:teuthology.misc.health.trial181.stderr:2026-02-26T03:25:12.561+0000 7f2987235640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:12.567 INFO:teuthology.misc.health.trial181.stderr:2026-02-26T03:25:12.566+0000 7f2987235640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:12.601 INFO:teuthology.misc.health.trial181.stdout: 2026-02-26T03:25:12.601 INFO:teuthology.misc.health.trial181.stdout:{"epoch":6,"fsid":"7aaab781-9a96-4919-9f35-646a2d4e5c22","created":"2026-02-26T03:25:10.083274+0000","modified":"2026-02-26T03:25:11.849219+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:25:11.849219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7965c363-a204-48ea-b372-753f691a6991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8435dd82-163b-48e9-9691-65b13227e0f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5918fd4e-82ea-4938-9231-f307a16b051c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:12.609 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:25:13.537 INFO:tasks.ceph.mgr.x.trial181.stderr:2026-02-26T03:25:13.536+0000 7f6304514640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:25:18.911 DEBUG:teuthology.orchestra.run.trial181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:19.006 INFO:teuthology.misc.health.trial181.stderr:2026-02-26T03:25:19.000+0000 7ff97ab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:19.010 INFO:teuthology.misc.health.trial181.stderr:2026-02-26T03:25:19.008+0000 7ff97ab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:19.143 INFO:teuthology.misc.health.trial181.stdout: 2026-02-26T03:25:19.143 INFO:teuthology.misc.health.trial181.stdout:{"epoch":7,"fsid":"7aaab781-9a96-4919-9f35-646a2d4e5c22","created":"2026-02-26T03:25:10.083274+0000","modified":"2026-02-26T03:25:18.329015+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:25:11.849219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7965c363-a204-48ea-b372-753f691a6991","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8435dd82-163b-48e9-9691-65b13227e0f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5918fd4e-82ea-4938-9231-f307a16b051c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:19.157 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:25:25.459 DEBUG:teuthology.orchestra.run.trial181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:25.527 INFO:teuthology.misc.health.trial181.stderr:2026-02-26T03:25:25.524+0000 7f773bf62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:25.530 INFO:teuthology.misc.health.trial181.stderr:2026-02-26T03:25:25.528+0000 7f773bf62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:25.642 INFO:teuthology.misc.health.trial181.stdout: 2026-02-26T03:25:25.643 INFO:teuthology.misc.health.trial181.stdout:{"epoch":13,"fsid":"7aaab781-9a96-4919-9f35-646a2d4e5c22","created":"2026-02-26T03:25:10.083274+0000","modified":"2026-02-26T03:25:25.358317+0000","last_up_change":"2026-02-26T03:25:21.340065+0000","last_in_change":"2026-02-26T03:25:11.849219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:22.547856+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7965c363-a204-48ea-b372-753f691a6991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6803","nonce":1119042789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6802","nonce":1119042789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6809","nonce":1119042789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6812","nonce":1119042789}]},"public_addr":"10.20.193.181:6803/1119042789","cluster_addr":"10.20.193.181:6802/1119042789","heartbeat_back_addr":"10.20.193.181:6809/1119042789","heartbeat_front_addr":"10.20.193.181:6812/1119042789","state":["exists","up"]},{"osd":1,"uuid":"8435dd82-163b-48e9-9691-65b13227e0f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6805","nonce":2324789186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6804","nonce":2324789186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6806","nonce":2324789186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6807","nonce":2324789186}]},"public_addr":"10.20.193.181:6805/2324789186","cluster_addr":"10.20.193.181:6804/2324789186","heartbeat_back_addr":"10.20.193.181:6806/2324789186","heartbeat_front_addr":"10.20.193.181:6807/2324789186","state":["exists","up"]},{"osd":2,"uuid":"5918fd4e-82ea-4938-9231-f307a16b051c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6811","nonce":2775741130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6810","nonce":2775741130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6813","nonce":2775741130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6814","nonce":2775741130}]},"public_addr":"10.20.193.181:6811/2775741130","cluster_addr":"10.20.193.181:6810/2775741130","heartbeat_back_addr":"10.20.193.181:6813/2775741130","heartbeat_front_addr":"10.20.193.181:6814/2775741130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:25.651 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:25:25.651 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:25:25.651 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:25:25.726 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:25.721+0000 7efdf6051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:25.728 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:25.727+0000 7efdf6051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:26.362 INFO:teuthology.orchestra.run.trial181.stderr:pool 'rbd' created 2026-02-26T03:25:26.375 DEBUG:teuthology.orchestra.run.trial181:> rbd --cluster ceph pool init rbd 2026-02-26T03:25:26.421 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:26.419+0000 7fc46f210240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:26.421 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:26.419+0000 7fc46f210240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:26.424 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:26.422+0000 7fc46f210240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.405 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:25:29.406 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-26T03:25:29.406 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:25:29.482 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:29.478+0000 7f327821a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.484 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:29.483+0000 7f327821a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.597 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:29.607 INFO:teuthology.orchestra.run.trial181.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-26T03:25:29.607 INFO:tasks.ceph_manager:config epoch is 1 2026-02-26T03:25:29.607 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:25:29.607 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:25:29.608 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:25:29.682 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:29.679+0000 7f2cab251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.684 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:29.683+0000 7f2cab251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.830 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:29.838 INFO:teuthology.orchestra.run.trial181.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6800","nonce":2498087835},{"type":"v1","addr":"10.20.193.181:6801","nonce":2498087835}]},"active_addr":"10.20.193.181:6801/2498087835","active_change":"2026-02-26T03:25:12.522427+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.181:0","nonce":4258714446}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.181:0","nonce":991573605}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.181:0","nonce":3585830944}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.181:0","nonce":3831462700}]}]} 2026-02-26T03:25:29.843 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:25:29.843 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:25:29.843 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:29.920 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:29.916+0000 7f3a6af0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:29.922 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:29.920+0000 7f3a6af0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.036 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:30.036 INFO:teuthology.orchestra.run.trial181.stdout:{"epoch":17,"fsid":"7aaab781-9a96-4919-9f35-646a2d4e5c22","created":"2026-02-26T03:25:10.083274+0000","modified":"2026-02-26T03:25:29.373312+0000","last_up_change":"2026-02-26T03:25:21.340065+0000","last_in_change":"2026-02-26T03:25:11.849219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:22.547856+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:25:25.843987+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7965c363-a204-48ea-b372-753f691a6991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6803","nonce":1119042789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6802","nonce":1119042789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6809","nonce":1119042789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6812","nonce":1119042789}]},"public_addr":"10.20.193.181:6803/1119042789","cluster_addr":"10.20.193.181:6802/1119042789","heartbeat_back_addr":"10.20.193.181:6809/1119042789","heartbeat_front_addr":"10.20.193.181:6812/1119042789","state":["exists","up"]},{"osd":1,"uuid":"8435dd82-163b-48e9-9691-65b13227e0f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6805","nonce":2324789186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6804","nonce":2324789186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6806","nonce":2324789186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6807","nonce":2324789186}]},"public_addr":"10.20.193.181:6805/2324789186","cluster_addr":"10.20.193.181:6804/2324789186","heartbeat_back_addr":"10.20.193.181:6806/2324789186","heartbeat_front_addr":"10.20.193.181:6807/2324789186","state":["exists","up"]},{"osd":2,"uuid":"5918fd4e-82ea-4938-9231-f307a16b051c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6811","nonce":2775741130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6810","nonce":2775741130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6813","nonce":2775741130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6814","nonce":2775741130}]},"public_addr":"10.20.193.181:6811/2775741130","cluster_addr":"10.20.193.181:6810/2775741130","heartbeat_back_addr":"10.20.193.181:6813/2775741130","heartbeat_front_addr":"10.20.193.181:6814/2775741130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:30.046 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:25:30.046 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:25:30.124 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.120+0000 7f997630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.127 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.125+0000 7f997630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.239 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:30.239 INFO:teuthology.orchestra.run.trial181.stdout:{"epoch":17,"fsid":"7aaab781-9a96-4919-9f35-646a2d4e5c22","created":"2026-02-26T03:25:10.083274+0000","modified":"2026-02-26T03:25:29.373312+0000","last_up_change":"2026-02-26T03:25:21.340065+0000","last_in_change":"2026-02-26T03:25:11.849219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:22.547856+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:25:25.843987+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7965c363-a204-48ea-b372-753f691a6991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6803","nonce":1119042789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6802","nonce":1119042789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6809","nonce":1119042789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6812","nonce":1119042789}]},"public_addr":"10.20.193.181:6803/1119042789","cluster_addr":"10.20.193.181:6802/1119042789","heartbeat_back_addr":"10.20.193.181:6809/1119042789","heartbeat_front_addr":"10.20.193.181:6812/1119042789","state":["exists","up"]},{"osd":1,"uuid":"8435dd82-163b-48e9-9691-65b13227e0f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6805","nonce":2324789186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6804","nonce":2324789186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6806","nonce":2324789186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6807","nonce":2324789186}]},"public_addr":"10.20.193.181:6805/2324789186","cluster_addr":"10.20.193.181:6804/2324789186","heartbeat_back_addr":"10.20.193.181:6806/2324789186","heartbeat_front_addr":"10.20.193.181:6807/2324789186","state":["exists","up"]},{"osd":2,"uuid":"5918fd4e-82ea-4938-9231-f307a16b051c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6811","nonce":2775741130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6810","nonce":2775741130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6813","nonce":2775741130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6814","nonce":2775741130}]},"public_addr":"10.20.193.181:6811/2775741130","cluster_addr":"10.20.193.181:6810/2775741130","heartbeat_back_addr":"10.20.193.181:6813/2775741130","heartbeat_front_addr":"10.20.193.181:6814/2775741130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:25:30.249 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:25:30.250 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:25:30.250 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:25:30.330 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.325+0000 7f8ff0a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.330 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.325+0000 7f79acc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.331 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.327+0000 7fc0abb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.332 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.331+0000 7f8ff0a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.332 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.331+0000 7f79acc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.333 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.332+0000 7fc0abb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.349 INFO:teuthology.orchestra.run.trial181.stdout:38654705673 2026-02-26T03:25:30.349 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:25:30.350 INFO:teuthology.orchestra.run.trial181.stdout:42949672968 2026-02-26T03:25:30.351 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:25:30.352 INFO:teuthology.orchestra.run.trial181.stdout:42949672968 2026-02-26T03:25:30.352 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:25:30.470 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.465+0000 7f4fcc25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.470 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.465+0000 7f050390b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.471 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.466+0000 7ff0f7f8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.472 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.470+0000 7f050390b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.472 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.471+0000 7f4fcc25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.473 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:30.471+0000 7ff0f7f8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:30.589 INFO:teuthology.orchestra.run.trial181.stdout:42949672965 2026-02-26T03:25:30.594 INFO:teuthology.orchestra.run.trial181.stdout:38654705669 2026-02-26T03:25:30.595 INFO:teuthology.orchestra.run.trial181.stdout:42949672965 2026-02-26T03:25:30.600 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672965 for osd.2 2026-02-26T03:25:30.604 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705669 for osd.1 2026-02-26T03:25:30.607 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672965 for osd.0 2026-02-26T03:25:31.601 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:25:31.605 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:25:31.608 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:25:31.681 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.676+0000 7f3bea2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:31.683 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.682+0000 7f3bea2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:31.723 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.718+0000 7fb9e4b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:31.725 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.721+0000 7fd95645a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:31.725 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.724+0000 7fb9e4b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:31.727 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.725+0000 7fd95645a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:31.798 INFO:teuthology.orchestra.run.trial181.stdout:42949672968 2026-02-26T03:25:31.809 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.2 2026-02-26T03:25:31.810 DEBUG:teuthology.parallel:result is None 2026-02-26T03:25:31.845 INFO:teuthology.orchestra.run.trial181.stdout:38654705674 2026-02-26T03:25:31.845 INFO:teuthology.orchestra.run.trial181.stdout:42949672968 2026-02-26T03:25:31.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.0 2026-02-26T03:25:31.858 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705674 for osd.1 2026-02-26T03:25:31.858 DEBUG:teuthology.parallel:result is None 2026-02-26T03:25:31.858 DEBUG:teuthology.parallel:result is None 2026-02-26T03:25:31.858 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:25:31.859 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:25:31.974 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.970+0000 7f23cfd6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:31.976 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:31.975+0000 7f23cfd6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:32.087 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:32.087 INFO:teuthology.orchestra.run.trial181.stderr:dumped all 2026-02-26T03:25:32.096 INFO:teuthology.orchestra.run.trial181.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-26T03:25:30.535637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2524,"kb_used_data":2524,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497444,"statfs":{"total":160255967232,"available":160253382656,"internally_reserved":0,"allocated":2584576,"data_stored":2584576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.170044"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.388578+0000","last_change":"2026-02-26T03:25:29.391728+0000","last_active":"2026-02-26T03:25:29.388578+0000","last_peered":"2026-02-26T03:25:29.388578+0000","last_clean":"2026-02-26T03:25:29.388578+0000","last_became_active":"2026-02-26T03:25:27.412247+0000","last_became_peered":"2026-02-26T03:25:27.412247+0000","last_unstale":"2026-02-26T03:25:29.388578+0000","last_undegraded":"2026-02-26T03:25:29.388578+0000","last_fullsized":"2026-02-26T03:25:29.388578+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391023+0000","last_change":"2026-02-26T03:25:29.393057+0000","last_active":"2026-02-26T03:25:29.391023+0000","last_peered":"2026-02-26T03:25:29.391023+0000","last_clean":"2026-02-26T03:25:29.391023+0000","last_became_active":"2026-02-26T03:25:27.415950+0000","last_became_peered":"2026-02-26T03:25:27.415950+0000","last_unstale":"2026-02-26T03:25:29.391023+0000","last_undegraded":"2026-02-26T03:25:29.391023+0000","last_fullsized":"2026-02-26T03:25:29.391023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.390285+0000","last_change":"2026-02-26T03:25:29.394430+0000","last_active":"2026-02-26T03:25:29.390285+0000","last_peered":"2026-02-26T03:25:29.390285+0000","last_clean":"2026-02-26T03:25:29.390285+0000","last_became_active":"2026-02-26T03:25:27.421547+0000","last_became_peered":"2026-02-26T03:25:27.421547+0000","last_unstale":"2026-02-26T03:25:29.390285+0000","last_undegraded":"2026-02-26T03:25:29.390285+0000","last_fullsized":"2026-02-26T03:25:29.390285+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391706+0000","last_change":"2026-02-26T03:25:29.394818+0000","last_active":"2026-02-26T03:25:29.391706+0000","last_peered":"2026-02-26T03:25:29.391706+0000","last_clean":"2026-02-26T03:25:29.391706+0000","last_became_active":"2026-02-26T03:25:27.419349+0000","last_became_peered":"2026-02-26T03:25:27.419349+0000","last_unstale":"2026-02-26T03:25:29.391706+0000","last_undegraded":"2026-02-26T03:25:29.391706+0000","last_fullsized":"2026-02-26T03:25:29.391706+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391070+0000","last_change":"2026-02-26T03:25:29.391070+0000","last_active":"2026-02-26T03:25:29.391070+0000","last_peered":"2026-02-26T03:25:29.391070+0000","last_clean":"2026-02-26T03:25:29.391070+0000","last_became_active":"2026-02-26T03:25:27.431933+0000","last_became_peered":"2026-02-26T03:25:27.431933+0000","last_unstale":"2026-02-26T03:25:29.391070+0000","last_undegraded":"2026-02-26T03:25:29.391070+0000","last_fullsized":"2026-02-26T03:25:29.391070+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.388523+0000","last_change":"2026-02-26T03:25:29.390939+0000","last_active":"2026-02-26T03:25:29.388523+0000","last_peered":"2026-02-26T03:25:29.388523+0000","last_clean":"2026-02-26T03:25:29.388523+0000","last_became_active":"2026-02-26T03:25:27.431293+0000","last_became_peered":"2026-02-26T03:25:27.431293+0000","last_unstale":"2026-02-26T03:25:29.388523+0000","last_undegraded":"2026-02-26T03:25:29.388523+0000","last_fullsized":"2026-02-26T03:25:29.388523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.386541+0000","last_change":"2026-02-26T03:25:29.389926+0000","last_active":"2026-02-26T03:25:29.386541+0000","last_peered":"2026-02-26T03:25:29.386541+0000","last_clean":"2026-02-26T03:25:29.386541+0000","last_became_active":"2026-02-26T03:25:27.436171+0000","last_became_peered":"2026-02-26T03:25:27.436171+0000","last_unstale":"2026-02-26T03:25:29.386541+0000","last_undegraded":"2026-02-26T03:25:29.386541+0000","last_fullsized":"2026-02-26T03:25:29.386541+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391010+0000","last_change":"2026-02-26T03:25:29.393046+0000","last_active":"2026-02-26T03:25:29.391010+0000","last_peered":"2026-02-26T03:25:29.391010+0000","last_clean":"2026-02-26T03:25:29.391010+0000","last_became_active":"2026-02-26T03:25:27.412304+0000","last_became_peered":"2026-02-26T03:25:27.412304+0000","last_unstale":"2026-02-26T03:25:29.391010+0000","last_undegraded":"2026-02-26T03:25:29.391010+0000","last_fullsized":"2026-02-26T03:25:29.391010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.388809+0000","last_change":"2026-02-26T03:25:24.393577+0000","last_active":"2026-02-26T03:25:29.388809+0000","last_peered":"2026-02-26T03:25:29.388809+0000","last_clean":"2026-02-26T03:25:29.388809+0000","last_became_active":"2026-02-26T03:25:24.384128+0000","last_became_peered":"2026-02-26T03:25:24.384128+0000","last_unstale":"2026-02-26T03:25:29.388809+0000","last_undegraded":"2026-02-26T03:25:29.388809+0000","last_fullsized":"2026-02-26T03:25:29.388809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_clean_scrub_stamp":"2026-02-26T03:25:23.349596+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:25:32.097 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:25:32.172 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:32.169+0000 7fcec89c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:32.174 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:32.173+0000 7fcec89c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:32.286 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:32.286 INFO:teuthology.orchestra.run.trial181.stderr:dumped all 2026-02-26T03:25:32.294 INFO:teuthology.orchestra.run.trial181.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-26T03:25:30.535637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2524,"kb_used_data":2524,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497444,"statfs":{"total":160255967232,"available":160253382656,"internally_reserved":0,"allocated":2584576,"data_stored":2584576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.170044"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.388578+0000","last_change":"2026-02-26T03:25:29.391728+0000","last_active":"2026-02-26T03:25:29.388578+0000","last_peered":"2026-02-26T03:25:29.388578+0000","last_clean":"2026-02-26T03:25:29.388578+0000","last_became_active":"2026-02-26T03:25:27.412247+0000","last_became_peered":"2026-02-26T03:25:27.412247+0000","last_unstale":"2026-02-26T03:25:29.388578+0000","last_undegraded":"2026-02-26T03:25:29.388578+0000","last_fullsized":"2026-02-26T03:25:29.388578+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391023+0000","last_change":"2026-02-26T03:25:29.393057+0000","last_active":"2026-02-26T03:25:29.391023+0000","last_peered":"2026-02-26T03:25:29.391023+0000","last_clean":"2026-02-26T03:25:29.391023+0000","last_became_active":"2026-02-26T03:25:27.415950+0000","last_became_peered":"2026-02-26T03:25:27.415950+0000","last_unstale":"2026-02-26T03:25:29.391023+0000","last_undegraded":"2026-02-26T03:25:29.391023+0000","last_fullsized":"2026-02-26T03:25:29.391023+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.390285+0000","last_change":"2026-02-26T03:25:29.394430+0000","last_active":"2026-02-26T03:25:29.390285+0000","last_peered":"2026-02-26T03:25:29.390285+0000","last_clean":"2026-02-26T03:25:29.390285+0000","last_became_active":"2026-02-26T03:25:27.421547+0000","last_became_peered":"2026-02-26T03:25:27.421547+0000","last_unstale":"2026-02-26T03:25:29.390285+0000","last_undegraded":"2026-02-26T03:25:29.390285+0000","last_fullsized":"2026-02-26T03:25:29.390285+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391706+0000","last_change":"2026-02-26T03:25:29.394818+0000","last_active":"2026-02-26T03:25:29.391706+0000","last_peered":"2026-02-26T03:25:29.391706+0000","last_clean":"2026-02-26T03:25:29.391706+0000","last_became_active":"2026-02-26T03:25:27.419349+0000","last_became_peered":"2026-02-26T03:25:27.419349+0000","last_unstale":"2026-02-26T03:25:29.391706+0000","last_undegraded":"2026-02-26T03:25:29.391706+0000","last_fullsized":"2026-02-26T03:25:29.391706+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391070+0000","last_change":"2026-02-26T03:25:29.391070+0000","last_active":"2026-02-26T03:25:29.391070+0000","last_peered":"2026-02-26T03:25:29.391070+0000","last_clean":"2026-02-26T03:25:29.391070+0000","last_became_active":"2026-02-26T03:25:27.431933+0000","last_became_peered":"2026-02-26T03:25:27.431933+0000","last_unstale":"2026-02-26T03:25:29.391070+0000","last_undegraded":"2026-02-26T03:25:29.391070+0000","last_fullsized":"2026-02-26T03:25:29.391070+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.388523+0000","last_change":"2026-02-26T03:25:29.390939+0000","last_active":"2026-02-26T03:25:29.388523+0000","last_peered":"2026-02-26T03:25:29.388523+0000","last_clean":"2026-02-26T03:25:29.388523+0000","last_became_active":"2026-02-26T03:25:27.431293+0000","last_became_peered":"2026-02-26T03:25:27.431293+0000","last_unstale":"2026-02-26T03:25:29.388523+0000","last_undegraded":"2026-02-26T03:25:29.388523+0000","last_fullsized":"2026-02-26T03:25:29.388523+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.386541+0000","last_change":"2026-02-26T03:25:29.389926+0000","last_active":"2026-02-26T03:25:29.386541+0000","last_peered":"2026-02-26T03:25:29.386541+0000","last_clean":"2026-02-26T03:25:29.386541+0000","last_became_active":"2026-02-26T03:25:27.436171+0000","last_became_peered":"2026-02-26T03:25:27.436171+0000","last_unstale":"2026-02-26T03:25:29.386541+0000","last_undegraded":"2026-02-26T03:25:29.386541+0000","last_fullsized":"2026-02-26T03:25:29.386541+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.391010+0000","last_change":"2026-02-26T03:25:29.393046+0000","last_active":"2026-02-26T03:25:29.391010+0000","last_peered":"2026-02-26T03:25:29.391010+0000","last_clean":"2026-02-26T03:25:29.391010+0000","last_became_active":"2026-02-26T03:25:27.412304+0000","last_became_peered":"2026-02-26T03:25:27.412304+0000","last_unstale":"2026-02-26T03:25:29.391010+0000","last_undegraded":"2026-02-26T03:25:29.391010+0000","last_fullsized":"2026-02-26T03:25:29.391010+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:25:29.388809+0000","last_change":"2026-02-26T03:25:24.393577+0000","last_active":"2026-02-26T03:25:29.388809+0000","last_peered":"2026-02-26T03:25:29.388809+0000","last_clean":"2026-02-26T03:25:29.388809+0000","last_became_active":"2026-02-26T03:25:24.384128+0000","last_became_peered":"2026-02-26T03:25:24.384128+0000","last_unstale":"2026-02-26T03:25:29.388809+0000","last_undegraded":"2026-02-26T03:25:29.388809+0000","last_fullsized":"2026-02-26T03:25:29.388809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_clean_scrub_stamp":"2026-02-26T03:25:23.349596+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:25:32.296 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:25:32.296 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:25:32.296 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:25:32.296 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:25:32.371 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:32.369+0000 7fa9b010b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:32.374 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:25:32.372+0000 7fa9b010b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:25:32.512 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:25:32.513 INFO:teuthology.orchestra.run.trial181.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:25:32.523 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:25:32.524 INFO:teuthology.run_tasks:Running task workunit... 2026-02-26T03:25:32.543 INFO:tasks.workunit:Pulling workunits from ref 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 2026-02-26T03:25:32.543 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-26T03:25:32.544 DEBUG:teuthology.orchestra.run.trial181:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:25:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:25:32.562 INFO:teuthology.orchestra.run.trial181.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-26T03:25:32.562 DEBUG:teuthology.orchestra.run.trial181:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:25:32.618 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-26T03:25:32.619 DEBUG:teuthology.orchestra.run.trial181:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-26T03:25:32.675 INFO:tasks.workunit:timeout=3h 2026-02-26T03:25:32.675 INFO:tasks.workunit:cleanup=True 2026-02-26T03:25:32.676 DEBUG:teuthology.orchestra.run.trial181:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 2026-02-26T03:25:32.738 INFO:tasks.workunit.client.0.trial181.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-26T03:25:56.322 INFO:tasks.workunit.client.0.trial181.stderr:Note: switching to '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'. 2026-02-26T03:25:56.322 INFO:tasks.workunit.client.0.trial181.stderr: 2026-02-26T03:25:56.322 INFO:tasks.workunit.client.0.trial181.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-26T03:25:56.322 INFO:tasks.workunit.client.0.trial181.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-26T03:25:56.322 INFO:tasks.workunit.client.0.trial181.stderr:state without impacting any branches by switching back to a branch. 2026-02-26T03:25:56.322 INFO:tasks.workunit.client.0.trial181.stderr: 2026-02-26T03:25:56.322 INFO:tasks.workunit.client.0.trial181.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr: 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr: git switch -c 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr: 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr:Or undo this operation with: 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr: 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr: git switch - 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr: 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr: 2026-02-26T03:25:56.323 INFO:tasks.workunit.client.0.trial181.stderr:HEAD is now at 1f87b69b1a8 Merge pull request #67522 from rhcs-dashboard/notification-banner-simple-mode 2026-02-26T03:25:56.325 DEBUG:teuthology.orchestra.run.trial181:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-26T03:25:56.382 INFO:tasks.workunit.client.0.trial181.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-26T03:25:56.384 INFO:tasks.workunit.client.0.trial181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T03:25:56.384 INFO:tasks.workunit.client.0.trial181.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-26T03:25:56.474 INFO:tasks.workunit.client.0.trial181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-26T03:25:56.506 INFO:tasks.workunit.client.0.trial181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-26T03:25:56.533 INFO:tasks.workunit.client.0.trial181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2026-02-26T03:25:56.535 INFO:tasks.workunit.client.0.trial181.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T03:25:56.535 INFO:tasks.workunit.client.0.trial181.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-26T03:25:56.560 INFO:tasks.workunit.client.0.trial181.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2026-02-26T03:25:56.565 DEBUG:teuthology.orchestra.run.trial181:> set -ex 2026-02-26T03:25:56.565 DEBUG:teuthology.orchestra.run.trial181:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-26T03:25:56.622 INFO:tasks.workunit:Running workunits matching rbd/test_librbd_python.sh on client.0... 2026-02-26T03:25:56.625 INFO:tasks.workunit:Running workunit rbd/test_librbd_python.sh... 2026-02-26T03:25:56.625 DEBUG:teuthology.orchestra.run.trial181:workunit test rbd/test_librbd_python.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2026-02-26T03:25:56.683 INFO:tasks.workunit.client.0.trial181.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2026-02-26T03:25:56.683 INFO:tasks.workunit.client.0.trial181.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind 2026-02-26T03:25:56.684 INFO:tasks.workunit.client.0.trial181.stderr:+ '[' -n '' ']' 2026-02-26T03:25:56.684 INFO:tasks.workunit.client.0.trial181.stderr:+ python3 -m pytest -v /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/pybind/test_rbd.py 2026-02-26T03:25:56.757 INFO:tasks.workunit.client.0.trial181.stdout:============================= test session starts ============================== 2026-02-26T03:25:56.757 INFO:tasks.workunit.client.0.trial181.stdout:platform linux -- Python 3.9.25, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 2026-02-26T03:25:56.757 INFO:tasks.workunit.client.0.trial181.stdout:cachedir: .pytest_cache 2026-02-26T03:25:56.757 INFO:tasks.workunit.client.0.trial181.stdout:rootdir: /home/ubuntu/cephtest/clone.client.0/src/test/pybind, configfile: pytest.ini 2026-02-26T03:25:57.050 INFO:tasks.workunit.client.0.trial181.stdout:collecting ... collected 196 items 2026-02-26T03:25:57.051 INFO:tasks.workunit.client.0.trial181.stdout: 2026-02-26T03:26:00.138 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_version PASSED [ 0%] 2026-02-26T03:26:00.650 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create PASSED [ 1%] 2026-02-26T03:26:05.111 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_defaults PASSED [ 1%] 2026-02-26T03:26:05.812 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_context_manager PASSED [ 2%] 2026-02-26T03:26:07.659 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_read_only PASSED [ 2%] 2026-02-26T03:26:08.778 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_dne PASSED [ 3%] 2026-02-26T03:26:09.927 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_readonly_dne PASSED [ 3%] 2026-02-26T03:26:09.927 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_open_by_id SKIPPED [ 4%] 2026-02-26T03:26:10.514 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_aio_open PASSED [ 4%] 2026-02-26T03:26:10.576 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_dne PASSED [ 5%] 2026-02-26T03:26:10.585 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_empty PASSED [ 5%] 2026-02-26T03:26:11.156 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list PASSED [ 6%] 2026-02-26T03:26:11.657 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress PASSED [ 6%] 2026-02-26T03:26:12.297 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_canceled PASSED [ 7%] 2026-02-26T03:26:12.786 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_remove_with_progress_except PASSED [ 7%] 2026-02-26T03:26:13.609 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename PASSED [ 8%] 2026-02-26T03:26:15.430 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_metadata PASSED [ 8%] 2026-02-26T03:26:15.479 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_config_list PASSED [ 9%] 2026-02-26T03:26:15.514 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_config_set_and_get_and_remove PASSED [ 9%] 2026-02-26T03:26:15.699 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_namespaces PASSED [ 10%] 2026-02-26T03:26:15.699 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_pool_stats SKIPPED [ 10%] 2026-02-26T03:26:15.700 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_to_string SKIPPED [ 11%] 2026-02-26T03:26:15.700 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_features_from_string SKIPPED [ 11%] 2026-02-26T03:26:16.286 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_update_features SKIPPED [ 12%] 2026-02-26T03:26:16.867 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_with_params SKIPPED [ 12%] 2026-02-26T03:26:17.423 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_id SKIPPED [ 13%] 2026-02-26T03:26:18.002 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_block_name_prefix PASSED [ 13%] 2026-02-26T03:26:18.565 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_data_pool_id PASSED [ 14%] 2026-02-26T03:26:19.127 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_timestamp PASSED [ 14%] 2026-02-26T03:26:19.695 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_access_timestamp PASSED [ 15%] 2026-02-26T03:26:20.280 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_modify_timestamp PASSED [ 15%] 2026-02-26T03:26:20.957 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_invalidate_cache PASSED [ 16%] 2026-02-26T03:26:21.526 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_stat PASSED [ 16%] 2026-02-26T03:26:22.107 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_flags PASSED [ 17%] 2026-02-26T03:26:22.770 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_auto_close PASSED [ 17%] 2026-02-26T03:26:23.362 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_use_after_close PASSED [ 18%] 2026-02-26T03:26:24.052 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write PASSED [ 18%] 2026-02-26T03:26:24.772 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_with_fadvise_flags PASSED [ 19%] 2026-02-26T03:26:25.485 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes PASSED [ 19%] 2026-02-26T03:26:26.218 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_zeroes_thick_provision PASSED [ 20%] 2026-02-26T03:26:26.801 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read PASSED [ 20%] 2026-02-26T03:26:27.374 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_with_fadvise_flags PASSED [ 21%] 2026-02-26T03:26:28.118 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_write PASSED [ 21%] 2026-02-26T03:26:28.703 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_large_read PASSED [ 22%] 2026-02-26T03:26:29.395 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_write_read PASSED [ 22%] 2026-02-26T03:26:29.972 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_read_bad_offset PASSED [ 23%] 2026-02-26T03:26:30.702 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize PASSED [ 23%] 2026-02-26T03:26:31.407 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_allow_shrink_False PASSED [ 24%] 2026-02-26T03:26:35.700 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_size PASSED [ 25%] 2026-02-26T03:26:36.489 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_down PASSED [ 25%] 2026-02-26T03:26:37.354 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_resize_bytes PASSED [ 26%] 2026-02-26T03:26:38.902 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy PASSED [ 26%] 2026-02-26T03:26:40.391 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy2 PASSED [ 27%] 2026-02-26T03:26:40.968 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_copy3 SKIPPED [ 27%] 2026-02-26T03:26:45.313 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy PASSED [ 28%] 2026-02-26T03:26:45.924 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v1 SKIPPED [ 28%] 2026-02-26T03:26:46.525 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_to_v2 SKIPPED [ 29%] 2026-02-26T03:26:47.131 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v1 SKIPPED [ 29%] 2026-02-26T03:26:47.741 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_to_v2 SKIPPED [ 30%] 2026-02-26T03:26:48.324 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v1_flatten SKIPPED [ 30%] 2026-02-26T03:26:48.900 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_deep_copy_clone_v2_flatten SKIPPED [ 31%] 2026-02-26T03:26:50.756 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap PASSED [ 31%] 2026-02-26T03:26:52.734 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_exists PASSED [ 32%] 2026-02-26T03:26:58.772 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_create_snap_flags PASSED [ 32%] 2026-02-26T03:27:02.782 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps PASSED [ 33%] 2026-02-26T03:27:04.781 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_snaps_iterator_auto_close PASSED [ 33%] 2026-02-26T03:27:06.834 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap PASSED [ 34%] 2026-02-26T03:27:07.432 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_not_found PASSED [ 34%] 2026-02-26T03:27:08.034 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap2 SKIPPED [ 35%] 2026-02-26T03:27:09.838 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_snap_by_id PASSED [ 35%] 2026-02-26T03:27:11.827 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rename_snap PASSED [ 36%] 2026-02-26T03:27:12.441 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_protect_snap SKIPPED [ 36%] 2026-02-26T03:27:14.831 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_exists PASSED [ 37%] 2026-02-26T03:27:16.849 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_timestamp PASSED [ 37%] 2026-02-26T03:27:24.868 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_limit_snaps PASSED [ 38%] 2026-02-26T03:27:25.483 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_exclusive_lock SKIPPED [ 38%] 2026-02-26T03:27:27.856 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_snap PASSED [ 39%] 2026-02-26T03:27:28.707 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_remove_with_watcher PASSED [ 39%] 2026-02-26T03:27:30.703 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap PASSED [ 40%] 2026-02-26T03:27:32.872 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_to_snap_sparse PASSED [ 40%] 2026-02-26T03:27:36.744 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_rollback_with_resize PASSED [ 41%] 2026-02-26T03:27:38.686 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap PASSED [ 41%] 2026-02-26T03:27:40.744 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_no_snap PASSED [ 42%] 2026-02-26T03:27:42.769 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_by_id PASSED [ 42%] 2026-02-26T03:27:48.736 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_name PASSED [ 43%] 2026-02-26T03:27:54.771 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_snap_get_id PASSED [ 43%] 2026-02-26T03:27:56.761 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_sparse PASSED [ 44%] 2026-02-26T03:36:44.840 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_many_snaps PASSED [ 44%] 2026-02-26T03:36:46.871 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_deleted PASSED [ 45%] 2026-02-26T03:36:50.844 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_set_snap_recreated PASSED [ 45%] 2026-02-26T03:36:51.563 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_lock_unlock PASSED [ 46%] 2026-02-26T03:36:53.141 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_list_lockers PASSED [ 46%] 2026-02-26T03:36:59.885 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate PASSED [ 47%] 2026-02-26T03:37:00.514 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate_from_trash_snap SKIPPED [ 47%] 2026-02-26T03:37:01.139 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_diff_iterate_exclude_parent SKIPPED [ 48%] 2026-02-26T03:37:01.762 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_read PASSED [ 48%] 2026-02-26T03:37:02.508 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write PASSED [ 49%] 2026-02-26T03:37:03.237 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_discard PASSED [ 50%] 2026-02-26T03:37:03.993 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_write_zeroes PASSED [ 50%] 2026-02-26T03:37:04.652 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_aio_flush PASSED [ 51%] 2026-02-26T03:37:10.846 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_metadata PASSED [ 51%] 2026-02-26T03:37:11.482 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_watchers_list PASSED [ 52%] 2026-02-26T03:37:12.374 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_config_list PASSED [ 52%] 2026-02-26T03:37:13.204 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_image_config_set_and_get_and_remove PASSED [ 53%] 2026-02-26T03:37:13.870 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_sparsify PASSED [ 53%] 2026-02-26T03:37:23.269 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks1 PASSED [ 54%] 2026-02-26T03:37:36.264 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImage::test_encryption_luks2 PASSED [ 54%] 2026-02-26T03:37:36.948 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_read PASSED [ 55%] 2026-02-26T03:37:37.772 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_write PASSED [ 55%] 2026-02-26T03:37:38.556 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestImageId::test_resize PASSED [ 56%] 2026-02-26T03:37:38.558 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params SKIPPED [ 56%] 2026-02-26T03:37:38.558 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params2 SKIPPED [ 57%] 2026-02-26T03:37:38.558 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_with_params3 SKIPPED [ 57%] 2026-02-26T03:37:38.559 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stripe_unit_and_count SKIPPED [ 58%] 2026-02-26T03:37:38.559 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_format SKIPPED [ 58%] 2026-02-26T03:37:38.559 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotected SKIPPED [ 59%] 2026-02-26T03:37:38.559 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_unprotect_with_children SKIPPED [ 59%] 2026-02-26T03:37:38.560 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_clone_by_snap_id SKIPPED [ 60%] 2026-02-26T03:37:38.560 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_stat SKIPPED [ 60%] 2026-02-26T03:37:38.560 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_stat SKIPPED [ 61%] 2026-02-26T03:37:38.560 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_io SKIPPED [ 61%] 2026-02-26T03:37:38.561 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_read SKIPPED [ 62%] 2026-02-26T03:37:38.561 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_write SKIPPED [ 62%] 2026-02-26T03:37:38.561 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_list_children SKIPPED [ 63%] 2026-02-26T03:37:38.562 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_errors SKIPPED [ 63%] 2026-02-26T03:37:38.562 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_basic SKIPPED [ 64%] 2026-02-26T03:37:38.562 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_smaller_order SKIPPED [ 64%] 2026-02-26T03:37:38.563 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_larger_order SKIPPED [ 65%] 2026-02-26T03:37:38.563 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_drops_cache SKIPPED [ 65%] 2026-02-26T03:37:38.563 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_multi_level SKIPPED [ 66%] 2026-02-26T03:37:38.564 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_flatten_with_progress SKIPPED [ 66%] 2026-02-26T03:37:38.564 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_resize_flatten_multi_level SKIPPED [ 67%] 2026-02-26T03:37:38.564 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_trash_snapshot SKIPPED [ 67%] 2026-02-26T03:37:38.565 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks1 SKIPPED [ 68%] 2026-02-26T03:37:38.565 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestClone::test_encryption_luks2 SKIPPED [ 68%] 2026-02-26T03:37:38.565 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_ownership SKIPPED [ 69%] 2026-02-26T03:37:38.566 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_snapshot_leadership SKIPPED [ 69%] 2026-02-26T03:37:38.566 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_read_only_leadership SKIPPED [ 70%] 2026-02-26T03:37:38.566 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_flatten SKIPPED [ 70%] 2026-02-26T03:37:38.567 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_resize SKIPPED [ 71%] 2026-02-26T03:37:38.567 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_create SKIPPED [ 71%] 2026-02-26T03:37:38.568 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_snap_rollback SKIPPED [ 72%] 2026-02-26T03:37:38.568 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_discard SKIPPED [ 72%] 2026-02-26T03:37:38.568 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_follower_write SKIPPED [ 73%] 2026-02-26T03:37:38.569 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_acquire_release_lock SKIPPED [ 73%] 2026-02-26T03:37:38.569 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestExclusiveLock::test_break_lock SKIPPED [ 74%] 2026-02-26T03:37:39.284 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_uuid PASSED [ 75%] 2026-02-26T03:37:39.974 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_site_name PASSED [ 75%] 2026-02-26T03:37:41.342 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_remote_namespace PASSED [ 76%] 2026-02-26T03:37:42.189 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer_bootstrap PASSED [ 76%] 2026-02-26T03:37:42.981 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_peer PASSED [ 77%] 2026-02-26T03:37:43.689 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image SKIPPED [ 77%] 2026-02-26T03:37:44.377 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_status SKIPPED [ 78%] 2026-02-26T03:37:52.064 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_mirror_image_create_snapshot PASSED [ 78%] 2026-02-26T03:37:56.569 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMirroring::test_aio_mirror_image_create_snapshot PASSED [ 79%] 2026-02-26T03:37:57.085 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_move PASSED [ 79%] 2026-02-26T03:37:58.099 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_purge PASSED [ 80%] 2026-02-26T03:37:58.652 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_denied PASSED [ 80%] 2026-02-26T03:37:59.157 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove PASSED [ 81%] 2026-02-26T03:37:59.668 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_remove_with_progress PASSED [ 81%] 2026-02-26T03:38:00.174 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_get PASSED [ 82%] 2026-02-26T03:38:01.218 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_list PASSED [ 82%] 2026-02-26T03:38:02.073 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestTrash::test_restore PASSED [ 83%] 2026-02-26T03:38:02.128 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_create_group PASSED [ 83%] 2026-02-26T03:38:02.214 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_rename_group PASSED [ 84%] 2026-02-26T03:38:02.218 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_empty PASSED [ 84%] 2026-02-26T03:38:02.272 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups PASSED [ 85%] 2026-02-26T03:38:02.325 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::test_list_groups_after_removed PASSED [ 85%] 2026-02-26T03:38:03.079 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_add PASSED [ 86%] 2026-02-26T03:38:03.794 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_empty PASSED [ 86%] 2026-02-26T03:38:04.568 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list PASSED [ 87%] 2026-02-26T03:38:05.499 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_move_to_trash PASSED [ 87%] 2026-02-26T03:38:07.069 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_list_remove PASSED [ 88%] 2026-02-26T03:38:07.745 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_get_id PASSED [ 88%] 2026-02-26T03:38:21.522 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_many_images PASSED [ 89%] 2026-02-26T03:38:22.493 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_image_remove PASSED [ 89%] 2026-02-26T03:38:25.510 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info PASSED [ 90%] 2026-02-26T03:38:26.236 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_get_info_no_member_images PASSED [ 90%] 2026-02-26T03:38:27.047 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap PASSED [ 91%] 2026-02-26T03:38:27.935 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_flags PASSED [ 91%] 2026-02-26T03:39:09.672 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_list_many PASSED [ 92%] 2026-02-26T03:39:12.060 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_namespace PASSED [ 92%] 2026-02-26T03:39:12.850 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rename PASSED [ 93%] 2026-02-26T03:39:13.537 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone SKIPPED [ 93%] 2026-02-26T03:39:14.248 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_clone_flatten SKIPPED [ 94%] 2026-02-26T03:39:15.983 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_diff_iterate PASSED [ 94%] 2026-02-26T03:39:30.170 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestGroups::test_group_snap_rollback PASSED [ 95%] 2026-02-26T03:39:32.470 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration PASSED [ 95%] 2026-02-26T03:39:32.471 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v1 SKIPPED [ 96%] 2026-02-26T03:39:32.471 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v1_to_v2 SKIPPED [ 96%] 2026-02-26T03:39:32.472 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v1 SKIPPED [ 97%] 2026-02-26T03:39:32.472 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_clone_v2_to_v2 SKIPPED [ 97%] 2026-02-26T03:39:38.388 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_import PASSED [ 98%] 2026-02-26T03:39:40.287 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migration_with_progress PASSED [ 98%] 2026-02-26T03:39:41.883 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort PASSED [ 99%] 2026-02-26T03:39:44.262 INFO:tasks.workunit.client.0.trial181.stdout:../../../clone.client.0/src/test/pybind/test_rbd.py::TestMigration::test_migrate_abort_with_progress PASSED [100%] 2026-02-26T03:39:44.262 INFO:tasks.workunit.client.0.trial181.stdout: 2026-02-26T03:39:44.262 INFO:tasks.workunit.client.0.trial181.stdout:=============================== warnings summary =============================== 2026-02-26T03:39:44.262 INFO:tasks.workunit.client.0.trial181.stdout:test_rbd.py::test_remove_with_progress_except 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored in: 'rbd.progress_callback' 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: Traceback (most recent call last): 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rbd.py", line 430, in progress_cb 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: raise Exception() 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: Exception 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) 2026-02-26T03:39:44.263 INFO:tasks.workunit.client.0.trial181.stdout: 2026-02-26T03:39:44.264 INFO:tasks.workunit.client.0.trial181.stdout:-- Docs: https://docs.pytest.org/en/stable/warnings.html 2026-02-26T03:39:44.264 INFO:tasks.workunit.client.0.trial181.stdout:============ 133 passed, 63 skipped, 1 warning in 827.50s (0:13:47) ============ 2026-02-26T03:39:44.307 INFO:tasks.workunit.client.0.trial181.stderr:+ exit 0 2026-02-26T03:39:44.307 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-26T03:39:44.308 DEBUG:teuthology.orchestra.run.trial181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-26T03:39:44.376 INFO:tasks.workunit:Stopping ['rbd/test_librbd_python.sh'] on client.0... 2026-02-26T03:39:44.376 DEBUG:teuthology.orchestra.run.trial181:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-26T03:39:44.784 DEBUG:teuthology.parallel:result is None 2026-02-26T03:39:44.785 DEBUG:teuthology.orchestra.run.trial181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:39:44.810 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:39:44.811 DEBUG:teuthology.orchestra.run.trial181:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-26T03:39:44.869 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T03:39:44.869 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:39:44.884 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:39:44.885 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:39:44.986 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:44.981+0000 7fd6aeddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:44.988 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:44.986+0000 7fd6aeddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:45.099 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:39:45.100 INFO:teuthology.orchestra.run.trial181.stderr:dumped all 2026-02-26T03:39:45.109 INFO:teuthology.orchestra.run.trial181.stdout:{"pg_ready":true,"pg_map":{"version":1028,"stamp":"2026-02-26T03:39:44.667926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":20504,"kb_used_data":20504,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156479464,"statfs":{"total":160255967232,"available":160234971136,"internally_reserved":0,"allocated":20996096,"data_stored":20996096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-14,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339943,"num_read_kb":-37734,"num_write":-8140,"num_write_kb":-41429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.267904+0000","last_change":"2026-02-26T03:25:29.391728+0000","last_active":"2026-02-26T03:39:37.267904+0000","last_peered":"2026-02-26T03:39:37.267904+0000","last_clean":"2026-02-26T03:39:37.267904+0000","last_became_active":"2026-02-26T03:25:27.412247+0000","last_became_peered":"2026-02-26T03:25:27.412247+0000","last_unstale":"2026-02-26T03:39:37.267904+0000","last_undegraded":"2026-02-26T03:39:37.267904+0000","last_fullsized":"2026-02-26T03:39:37.267904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.271072+0000","last_change":"2026-02-26T03:25:29.393057+0000","last_active":"2026-02-26T03:39:37.271072+0000","last_peered":"2026-02-26T03:39:37.271072+0000","last_clean":"2026-02-26T03:39:37.271072+0000","last_became_active":"2026-02-26T03:25:27.415950+0000","last_became_peered":"2026-02-26T03:25:27.415950+0000","last_unstale":"2026-02-26T03:39:37.271072+0000","last_undegraded":"2026-02-26T03:39:37.271072+0000","last_fullsized":"2026-02-26T03:39:37.271072+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.267603+0000","last_change":"2026-02-26T03:25:29.394430+0000","last_active":"2026-02-26T03:39:37.267603+0000","last_peered":"2026-02-26T03:39:37.267603+0000","last_clean":"2026-02-26T03:39:37.267603+0000","last_became_active":"2026-02-26T03:25:27.421547+0000","last_became_peered":"2026-02-26T03:25:27.421547+0000","last_unstale":"2026-02-26T03:39:37.267603+0000","last_undegraded":"2026-02-26T03:39:37.267603+0000","last_fullsized":"2026-02-26T03:39:37.267603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.272606+0000","last_change":"2026-02-26T03:25:29.394818+0000","last_active":"2026-02-26T03:39:37.272606+0000","last_peered":"2026-02-26T03:39:37.272606+0000","last_clean":"2026-02-26T03:39:37.272606+0000","last_became_active":"2026-02-26T03:25:27.419349+0000","last_became_peered":"2026-02-26T03:25:27.419349+0000","last_unstale":"2026-02-26T03:39:37.272606+0000","last_undegraded":"2026-02-26T03:39:37.272606+0000","last_fullsized":"2026-02-26T03:39:37.272606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1766,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.262390+0000","last_change":"2026-02-26T03:25:29.391070+0000","last_active":"2026-02-26T03:39:37.262390+0000","last_peered":"2026-02-26T03:39:37.262390+0000","last_clean":"2026-02-26T03:39:37.262390+0000","last_became_active":"2026-02-26T03:25:27.431933+0000","last_became_peered":"2026-02-26T03:25:27.431933+0000","last_unstale":"2026-02-26T03:39:37.262390+0000","last_undegraded":"2026-02-26T03:39:37.262390+0000","last_fullsized":"2026-02-26T03:39:37.262390+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.261185+0000","last_change":"2026-02-26T03:25:29.390939+0000","last_active":"2026-02-26T03:39:37.261185+0000","last_peered":"2026-02-26T03:39:37.261185+0000","last_clean":"2026-02-26T03:39:37.261185+0000","last_became_active":"2026-02-26T03:25:27.431293+0000","last_became_peered":"2026-02-26T03:25:27.431293+0000","last_unstale":"2026-02-26T03:39:37.261185+0000","last_undegraded":"2026-02-26T03:39:37.261185+0000","last_fullsized":"2026-02-26T03:39:37.261185+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.261194+0000","last_change":"2026-02-26T03:25:29.389926+0000","last_active":"2026-02-26T03:39:37.261194+0000","last_peered":"2026-02-26T03:39:37.261194+0000","last_clean":"2026-02-26T03:39:37.261194+0000","last_became_active":"2026-02-26T03:25:27.436171+0000","last_became_peered":"2026-02-26T03:25:27.436171+0000","last_unstale":"2026-02-26T03:39:37.261194+0000","last_undegraded":"2026-02-26T03:39:37.261194+0000","last_fullsized":"2026-02-26T03:39:37.261194+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1765,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.271427+0000","last_change":"2026-02-26T03:25:29.393046+0000","last_active":"2026-02-26T03:39:37.271427+0000","last_peered":"2026-02-26T03:39:37.271427+0000","last_clean":"2026-02-26T03:39:37.271427+0000","last_became_active":"2026-02-26T03:25:27.412304+0000","last_became_peered":"2026-02-26T03:25:27.412304+0000","last_unstale":"2026-02-26T03:39:37.271427+0000","last_undegraded":"2026-02-26T03:39:37.271427+0000","last_fullsized":"2026-02-26T03:39:37.271427+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1839,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.266486+0000","last_change":"2026-02-26T03:25:24.393577+0000","last_active":"2026-02-26T03:39:37.266486+0000","last_peered":"2026-02-26T03:39:37.266486+0000","last_clean":"2026-02-26T03:39:37.266486+0000","last_became_active":"2026-02-26T03:25:24.384128+0000","last_became_peered":"2026-02-26T03:25:24.384128+0000","last_unstale":"2026-02-26T03:39:37.266486+0000","last_undegraded":"2026-02-26T03:39:37.266486+0000","last_fullsized":"2026-02-26T03:39:37.266486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_clean_scrub_stamp":"2026-02-26T03:25:23.349596+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160235237376,"internally_reserved":0,"allocated":20729856,"data_stored":20729856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":5664,"kb_used_data":5664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52160992,"statfs":{"total":53418655744,"available":53412855808,"internally_reserved":0,"allocated":5799936,"data_stored":5799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":6924,"kb_used_data":6924,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159732,"statfs":{"total":53418655744,"available":53411565568,"internally_reserved":0,"allocated":7090176,"data_stored":7090176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":7916,"kb_used_data":7916,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52158740,"statfs":{"total":53418655744,"available":53410549760,"internally_reserved":0,"allocated":8105984,"data_stored":8105984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53411532800,"internally_reserved":0,"allocated":7122944,"data_stored":7122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53412818944,"internally_reserved":0,"allocated":5836800,"data_stored":5836800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:39:45.110 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:39:45.187 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:45.183+0000 7fae2848b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:45.189 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:45.187+0000 7fae2848b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:45.308 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:39:45.308 INFO:teuthology.orchestra.run.trial181.stderr:dumped all 2026-02-26T03:39:45.318 INFO:teuthology.orchestra.run.trial181.stdout:{"pg_ready":true,"pg_map":{"version":1029,"stamp":"2026-02-26T03:39:45.219444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":20504,"kb_used_data":20504,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156479464,"statfs":{"total":160255967232,"available":160234971136,"internally_reserved":0,"allocated":20996096,"data_stored":20996096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-14,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339943,"num_read_kb":-37734,"num_write":-8140,"num_write_kb":-41429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.971259"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.267904+0000","last_change":"2026-02-26T03:25:29.391728+0000","last_active":"2026-02-26T03:39:37.267904+0000","last_peered":"2026-02-26T03:39:37.267904+0000","last_clean":"2026-02-26T03:39:37.267904+0000","last_became_active":"2026-02-26T03:25:27.412247+0000","last_became_peered":"2026-02-26T03:25:27.412247+0000","last_unstale":"2026-02-26T03:39:37.267904+0000","last_undegraded":"2026-02-26T03:39:37.267904+0000","last_fullsized":"2026-02-26T03:39:37.267904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.271072+0000","last_change":"2026-02-26T03:25:29.393057+0000","last_active":"2026-02-26T03:39:37.271072+0000","last_peered":"2026-02-26T03:39:37.271072+0000","last_clean":"2026-02-26T03:39:37.271072+0000","last_became_active":"2026-02-26T03:25:27.415950+0000","last_became_peered":"2026-02-26T03:25:27.415950+0000","last_unstale":"2026-02-26T03:39:37.271072+0000","last_undegraded":"2026-02-26T03:39:37.271072+0000","last_fullsized":"2026-02-26T03:39:37.271072+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.267603+0000","last_change":"2026-02-26T03:25:29.394430+0000","last_active":"2026-02-26T03:39:37.267603+0000","last_peered":"2026-02-26T03:39:37.267603+0000","last_clean":"2026-02-26T03:39:37.267603+0000","last_became_active":"2026-02-26T03:25:27.421547+0000","last_became_peered":"2026-02-26T03:25:27.421547+0000","last_unstale":"2026-02-26T03:39:37.267603+0000","last_undegraded":"2026-02-26T03:39:37.267603+0000","last_fullsized":"2026-02-26T03:39:37.267603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.272606+0000","last_change":"2026-02-26T03:25:29.394818+0000","last_active":"2026-02-26T03:39:37.272606+0000","last_peered":"2026-02-26T03:39:37.272606+0000","last_clean":"2026-02-26T03:39:37.272606+0000","last_became_active":"2026-02-26T03:25:27.419349+0000","last_became_peered":"2026-02-26T03:25:27.419349+0000","last_unstale":"2026-02-26T03:39:37.272606+0000","last_undegraded":"2026-02-26T03:39:37.272606+0000","last_fullsized":"2026-02-26T03:39:37.272606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1766,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.262390+0000","last_change":"2026-02-26T03:25:29.391070+0000","last_active":"2026-02-26T03:39:37.262390+0000","last_peered":"2026-02-26T03:39:37.262390+0000","last_clean":"2026-02-26T03:39:37.262390+0000","last_became_active":"2026-02-26T03:25:27.431933+0000","last_became_peered":"2026-02-26T03:25:27.431933+0000","last_unstale":"2026-02-26T03:39:37.262390+0000","last_undegraded":"2026-02-26T03:39:37.262390+0000","last_fullsized":"2026-02-26T03:39:37.262390+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.261185+0000","last_change":"2026-02-26T03:25:29.390939+0000","last_active":"2026-02-26T03:39:37.261185+0000","last_peered":"2026-02-26T03:39:37.261185+0000","last_clean":"2026-02-26T03:39:37.261185+0000","last_became_active":"2026-02-26T03:25:27.431293+0000","last_became_peered":"2026-02-26T03:25:27.431293+0000","last_unstale":"2026-02-26T03:39:37.261185+0000","last_undegraded":"2026-02-26T03:39:37.261185+0000","last_fullsized":"2026-02-26T03:39:37.261185+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.261194+0000","last_change":"2026-02-26T03:25:29.389926+0000","last_active":"2026-02-26T03:39:37.261194+0000","last_peered":"2026-02-26T03:39:37.261194+0000","last_clean":"2026-02-26T03:39:37.261194+0000","last_became_active":"2026-02-26T03:25:27.436171+0000","last_became_peered":"2026-02-26T03:25:27.436171+0000","last_unstale":"2026-02-26T03:39:37.261194+0000","last_undegraded":"2026-02-26T03:39:37.261194+0000","last_fullsized":"2026-02-26T03:39:37.261194+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1765,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.271427+0000","last_change":"2026-02-26T03:25:29.393046+0000","last_active":"2026-02-26T03:39:37.271427+0000","last_peered":"2026-02-26T03:39:37.271427+0000","last_clean":"2026-02-26T03:39:37.271427+0000","last_became_active":"2026-02-26T03:25:27.412304+0000","last_became_peered":"2026-02-26T03:25:27.412304+0000","last_unstale":"2026-02-26T03:39:37.271427+0000","last_undegraded":"2026-02-26T03:39:37.271427+0000","last_fullsized":"2026-02-26T03:39:37.271427+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1839,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.266486+0000","last_change":"2026-02-26T03:25:24.393577+0000","last_active":"2026-02-26T03:39:37.266486+0000","last_peered":"2026-02-26T03:39:37.266486+0000","last_clean":"2026-02-26T03:39:37.266486+0000","last_became_active":"2026-02-26T03:25:24.384128+0000","last_became_peered":"2026-02-26T03:25:24.384128+0000","last_unstale":"2026-02-26T03:39:37.266486+0000","last_undegraded":"2026-02-26T03:39:37.266486+0000","last_fullsized":"2026-02-26T03:39:37.266486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_clean_scrub_stamp":"2026-02-26T03:25:23.349596+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160235237376,"internally_reserved":0,"allocated":20729856,"data_stored":20729856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":5664,"kb_used_data":5664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52160992,"statfs":{"total":53418655744,"available":53412855808,"internally_reserved":0,"allocated":5799936,"data_stored":5799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":6924,"kb_used_data":6924,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159732,"statfs":{"total":53418655744,"available":53411565568,"internally_reserved":0,"allocated":7090176,"data_stored":7090176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":7916,"kb_used_data":7916,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52158740,"statfs":{"total":53418655744,"available":53410549760,"internally_reserved":0,"allocated":8105984,"data_stored":8105984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53411532800,"internally_reserved":0,"allocated":7122944,"data_stored":7122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53412818944,"internally_reserved":0,"allocated":5836800,"data_stored":5836800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:39:45.320 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:39:45.321 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:39:45.401 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:45.397+0000 7fde588b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:45.403 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:45.401+0000 7fde588b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:45.515 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:39:45.515 INFO:teuthology.orchestra.run.trial181.stderr:dumped all 2026-02-26T03:39:45.523 INFO:teuthology.orchestra.run.trial181.stdout:{"pg_ready":true,"pg_map":{"version":1029,"stamp":"2026-02-26T03:39:45.219444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":20504,"kb_used_data":20504,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156479464,"statfs":{"total":160255967232,"available":160234971136,"internally_reserved":0,"allocated":20996096,"data_stored":20996096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-14,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339943,"num_read_kb":-37734,"num_write":-8140,"num_write_kb":-41429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.971259"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.267904+0000","last_change":"2026-02-26T03:25:29.391728+0000","last_active":"2026-02-26T03:39:37.267904+0000","last_peered":"2026-02-26T03:39:37.267904+0000","last_clean":"2026-02-26T03:39:37.267904+0000","last_became_active":"2026-02-26T03:25:27.412247+0000","last_became_peered":"2026-02-26T03:25:27.412247+0000","last_unstale":"2026-02-26T03:39:37.267904+0000","last_undegraded":"2026-02-26T03:39:37.267904+0000","last_fullsized":"2026-02-26T03:39:37.267904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.271072+0000","last_change":"2026-02-26T03:25:29.393057+0000","last_active":"2026-02-26T03:39:37.271072+0000","last_peered":"2026-02-26T03:39:37.271072+0000","last_clean":"2026-02-26T03:39:37.271072+0000","last_became_active":"2026-02-26T03:25:27.415950+0000","last_became_peered":"2026-02-26T03:25:27.415950+0000","last_unstale":"2026-02-26T03:39:37.271072+0000","last_undegraded":"2026-02-26T03:39:37.271072+0000","last_fullsized":"2026-02-26T03:39:37.271072+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.267603+0000","last_change":"2026-02-26T03:25:29.394430+0000","last_active":"2026-02-26T03:39:37.267603+0000","last_peered":"2026-02-26T03:39:37.267603+0000","last_clean":"2026-02-26T03:39:37.267603+0000","last_became_active":"2026-02-26T03:25:27.421547+0000","last_became_peered":"2026-02-26T03:25:27.421547+0000","last_unstale":"2026-02-26T03:39:37.267603+0000","last_undegraded":"2026-02-26T03:39:37.267603+0000","last_fullsized":"2026-02-26T03:39:37.267603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.272606+0000","last_change":"2026-02-26T03:25:29.394818+0000","last_active":"2026-02-26T03:39:37.272606+0000","last_peered":"2026-02-26T03:39:37.272606+0000","last_clean":"2026-02-26T03:39:37.272606+0000","last_became_active":"2026-02-26T03:25:27.419349+0000","last_became_peered":"2026-02-26T03:25:27.419349+0000","last_unstale":"2026-02-26T03:39:37.272606+0000","last_undegraded":"2026-02-26T03:39:37.272606+0000","last_fullsized":"2026-02-26T03:39:37.272606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1766,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.262390+0000","last_change":"2026-02-26T03:25:29.391070+0000","last_active":"2026-02-26T03:39:37.262390+0000","last_peered":"2026-02-26T03:39:37.262390+0000","last_clean":"2026-02-26T03:39:37.262390+0000","last_became_active":"2026-02-26T03:25:27.431933+0000","last_became_peered":"2026-02-26T03:25:27.431933+0000","last_unstale":"2026-02-26T03:39:37.262390+0000","last_undegraded":"2026-02-26T03:39:37.262390+0000","last_fullsized":"2026-02-26T03:39:37.262390+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.261185+0000","last_change":"2026-02-26T03:25:29.390939+0000","last_active":"2026-02-26T03:39:37.261185+0000","last_peered":"2026-02-26T03:39:37.261185+0000","last_clean":"2026-02-26T03:39:37.261185+0000","last_became_active":"2026-02-26T03:25:27.431293+0000","last_became_peered":"2026-02-26T03:25:27.431293+0000","last_unstale":"2026-02-26T03:39:37.261185+0000","last_undegraded":"2026-02-26T03:39:37.261185+0000","last_fullsized":"2026-02-26T03:39:37.261185+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1764,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.261194+0000","last_change":"2026-02-26T03:25:29.389926+0000","last_active":"2026-02-26T03:39:37.261194+0000","last_peered":"2026-02-26T03:39:37.261194+0000","last_clean":"2026-02-26T03:39:37.261194+0000","last_became_active":"2026-02-26T03:25:27.436171+0000","last_became_peered":"2026-02-26T03:25:27.436171+0000","last_unstale":"2026-02-26T03:39:37.261194+0000","last_undegraded":"2026-02-26T03:39:37.261194+0000","last_fullsized":"2026-02-26T03:39:37.261194+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1765,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.271427+0000","last_change":"2026-02-26T03:25:29.393046+0000","last_active":"2026-02-26T03:39:37.271427+0000","last_peered":"2026-02-26T03:39:37.271427+0000","last_clean":"2026-02-26T03:39:37.271427+0000","last_became_active":"2026-02-26T03:25:27.412304+0000","last_became_peered":"2026-02-26T03:25:27.412304+0000","last_unstale":"2026-02-26T03:39:37.271427+0000","last_undegraded":"2026-02-26T03:39:37.271427+0000","last_fullsized":"2026-02-26T03:39:37.271427+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1839,"reported_epoch":600,"state":"active+clean","last_fresh":"2026-02-26T03:39:37.266486+0000","last_change":"2026-02-26T03:25:24.393577+0000","last_active":"2026-02-26T03:39:37.266486+0000","last_peered":"2026-02-26T03:39:37.266486+0000","last_clean":"2026-02-26T03:39:37.266486+0000","last_became_active":"2026-02-26T03:25:24.384128+0000","last_became_peered":"2026-02-26T03:25:24.384128+0000","last_unstale":"2026-02-26T03:39:37.266486+0000","last_undegraded":"2026-02-26T03:39:37.266486+0000","last_fullsized":"2026-02-26T03:39:37.266486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_clean_scrub_stamp":"2026-02-26T03:25:23.349596+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160235237376,"internally_reserved":0,"allocated":20729856,"data_stored":20729856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":5664,"kb_used_data":5664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52160992,"statfs":{"total":53418655744,"available":53412855808,"internally_reserved":0,"allocated":5799936,"data_stored":5799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":6924,"kb_used_data":6924,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52159732,"statfs":{"total":53418655744,"available":53411565568,"internally_reserved":0,"allocated":7090176,"data_stored":7090176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":7916,"kb_used_data":7916,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52158740,"statfs":{"total":53418655744,"available":53410549760,"internally_reserved":0,"allocated":8105984,"data_stored":8105984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53411532800,"internally_reserved":0,"allocated":7122944,"data_stored":7122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53410885632,"internally_reserved":0,"allocated":7770112,"data_stored":7770112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53412818944,"internally_reserved":0,"allocated":5836800,"data_stored":5836800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:39:45.524 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:39:45.601 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:45.597+0000 7f46b77b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:45.603 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:45.601+0000 7f46b77b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:45.720 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:39:45.721 INFO:teuthology.orchestra.run.trial181.stdout:{"epoch":602,"fsid":"7aaab781-9a96-4919-9f35-646a2d4e5c22","created":"2026-02-26T03:25:10.083274+0000","modified":"2026-02-26T03:39:45.215292+0000","last_up_change":"2026-02-26T03:25:21.340065+0000","last_in_change":"2026-02-26T03:25:11.849219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:25:22.547856+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:25:25.843987+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7965c363-a204-48ea-b372-753f691a6991","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6803","nonce":1119042789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6802","nonce":1119042789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6809","nonce":1119042789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6812","nonce":1119042789}]},"public_addr":"10.20.193.181:6803/1119042789","cluster_addr":"10.20.193.181:6802/1119042789","heartbeat_back_addr":"10.20.193.181:6809/1119042789","heartbeat_front_addr":"10.20.193.181:6812/1119042789","state":["exists","up"]},{"osd":1,"uuid":"8435dd82-163b-48e9-9691-65b13227e0f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6805","nonce":2324789186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6804","nonce":2324789186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6806","nonce":2324789186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6807","nonce":2324789186}]},"public_addr":"10.20.193.181:6805/2324789186","cluster_addr":"10.20.193.181:6804/2324789186","heartbeat_back_addr":"10.20.193.181:6806/2324789186","heartbeat_front_addr":"10.20.193.181:6807/2324789186","state":["exists","up"]},{"osd":2,"uuid":"5918fd4e-82ea-4938-9231-f307a16b051c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6811","nonce":2775741130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6810","nonce":2775741130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6813","nonce":2775741130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.181:6814","nonce":2775741130}]},"public_addr":"10.20.193.181:6811/2775741130","cluster_addr":"10.20.193.181:6810/2775741130","heartbeat_back_addr":"10.20.193.181:6813/2775741130","heartbeat_front_addr":"10.20.193.181:6814/2775741130","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":302}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:39:46.732 INFO:tasks.ceph:Scrubbing osd.0 2026-02-26T03:39:46.732 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:39:46.812 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:46.806+0000 7fc8f6b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:46.815 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:46.813+0000 7fc8f6b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:46.824 INFO:teuthology.orchestra.run.trial181.stdout:{ 2026-02-26T03:39:46.824 INFO:teuthology.orchestra.run.trial181.stdout: "success": "" 2026-02-26T03:39:46.825 INFO:teuthology.orchestra.run.trial181.stdout:} 2026-02-26T03:39:46.832 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-26T03:39:46.909 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:46.905+0000 7fe26e406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:46.911 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:46.910+0000 7fe26e406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.025 INFO:teuthology.orchestra.run.trial181.stderr:instructed osd(s) 0 to deep-scrub 2026-02-26T03:39:47.036 INFO:tasks.ceph:Scrubbing osd.1 2026-02-26T03:39:47.036 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:39:47.114 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.107+0000 7f74bd6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.116 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.115+0000 7f74b7fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.125 INFO:teuthology.orchestra.run.trial181.stdout:{ 2026-02-26T03:39:47.125 INFO:teuthology.orchestra.run.trial181.stdout: "success": "" 2026-02-26T03:39:47.125 INFO:teuthology.orchestra.run.trial181.stdout:} 2026-02-26T03:39:47.133 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-26T03:39:47.209 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.205+0000 7f6208c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.212 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.210+0000 7f6208c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.323 INFO:teuthology.orchestra.run.trial181.stderr:instructed osd(s) 1 to deep-scrub 2026-02-26T03:39:47.335 INFO:tasks.ceph:Scrubbing osd.2 2026-02-26T03:39:47.335 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-26T03:39:47.413 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.407+0000 7f8f5a451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.415 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.414+0000 7f8f5a451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.424 INFO:teuthology.orchestra.run.trial181.stdout:{ 2026-02-26T03:39:47.425 INFO:teuthology.orchestra.run.trial181.stdout: "success": "" 2026-02-26T03:39:47.425 INFO:teuthology.orchestra.run.trial181.stdout:} 2026-02-26T03:39:47.430 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-26T03:39:47.508 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.503+0000 7fe6dd451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.510 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.508+0000 7fe6dd451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.621 INFO:teuthology.orchestra.run.trial181.stderr:instructed osd(s) 2 to deep-scrub 2026-02-26T03:39:47.632 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:39:47.708 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.704+0000 7f7993451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.710 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:39:47.708+0000 7f7993451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:39:47.822 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:39:47.822 INFO:teuthology.orchestra.run.trial181.stderr:dumped all 2026-02-26T03:39:47.831 INFO:teuthology.orchestra.run.trial181.stdout:{"pg_ready":true,"pg_map":{"version":1030,"stamp":"2026-02-26T03:39:46.668256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":7692,"kb_used_data":7692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156492276,"statfs":{"total":160255967232,"available":160248090624,"internally_reserved":0,"allocated":7876608,"data_stored":7876608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199,"num_objects":-14,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":-339943,"num_read_kb":-37734,"num_write":-8140,"num_write_kb":-41429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001323"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1770,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.229327+0000","last_change":"2026-02-26T03:25:29.391728+0000","last_active":"2026-02-26T03:39:45.229327+0000","last_peered":"2026-02-26T03:39:45.229327+0000","last_clean":"2026-02-26T03:39:45.229327+0000","last_became_active":"2026-02-26T03:25:27.412247+0000","last_became_peered":"2026-02-26T03:25:27.412247+0000","last_unstale":"2026-02-26T03:39:45.229327+0000","last_undegraded":"2026-02-26T03:39:45.229327+0000","last_fullsized":"2026-02-26T03:39:45.229327+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1770,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.230578+0000","last_change":"2026-02-26T03:25:29.393057+0000","last_active":"2026-02-26T03:39:45.230578+0000","last_peered":"2026-02-26T03:39:45.230578+0000","last_clean":"2026-02-26T03:39:45.230578+0000","last_became_active":"2026-02-26T03:25:27.415950+0000","last_became_peered":"2026-02-26T03:25:27.415950+0000","last_unstale":"2026-02-26T03:39:45.230578+0000","last_undegraded":"2026-02-26T03:39:45.230578+0000","last_fullsized":"2026-02-26T03:39:45.230578+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1770,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.231472+0000","last_change":"2026-02-26T03:25:29.394430+0000","last_active":"2026-02-26T03:39:45.231472+0000","last_peered":"2026-02-26T03:39:45.231472+0000","last_clean":"2026-02-26T03:39:45.231472+0000","last_became_active":"2026-02-26T03:25:27.421547+0000","last_became_peered":"2026-02-26T03:25:27.421547+0000","last_unstale":"2026-02-26T03:39:45.231472+0000","last_undegraded":"2026-02-26T03:39:45.231472+0000","last_fullsized":"2026-02-26T03:39:45.231472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1770,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.231114+0000","last_change":"2026-02-26T03:25:29.394818+0000","last_active":"2026-02-26T03:39:45.231114+0000","last_peered":"2026-02-26T03:39:45.231114+0000","last_clean":"2026-02-26T03:39:45.231114+0000","last_became_active":"2026-02-26T03:25:27.419349+0000","last_became_peered":"2026-02-26T03:25:27.419349+0000","last_unstale":"2026-02-26T03:39:45.231114+0000","last_undegraded":"2026-02-26T03:39:45.231114+0000","last_fullsized":"2026-02-26T03:39:45.231114+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1772,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.230345+0000","last_change":"2026-02-26T03:25:29.391070+0000","last_active":"2026-02-26T03:39:45.230345+0000","last_peered":"2026-02-26T03:39:45.230345+0000","last_clean":"2026-02-26T03:39:45.230345+0000","last_became_active":"2026-02-26T03:25:27.431933+0000","last_became_peered":"2026-02-26T03:25:27.431933+0000","last_unstale":"2026-02-26T03:39:45.230345+0000","last_undegraded":"2026-02-26T03:39:45.230345+0000","last_fullsized":"2026-02-26T03:39:45.230345+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1770,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.231380+0000","last_change":"2026-02-26T03:25:29.390939+0000","last_active":"2026-02-26T03:39:45.231380+0000","last_peered":"2026-02-26T03:39:45.231380+0000","last_clean":"2026-02-26T03:39:45.231380+0000","last_became_active":"2026-02-26T03:25:27.431293+0000","last_became_peered":"2026-02-26T03:25:27.431293+0000","last_unstale":"2026-02-26T03:39:45.231380+0000","last_undegraded":"2026-02-26T03:39:45.231380+0000","last_fullsized":"2026-02-26T03:39:45.231380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1770,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.230701+0000","last_change":"2026-02-26T03:25:29.389926+0000","last_active":"2026-02-26T03:39:45.230701+0000","last_peered":"2026-02-26T03:39:45.230701+0000","last_clean":"2026-02-26T03:39:45.230701+0000","last_became_active":"2026-02-26T03:25:27.436171+0000","last_became_peered":"2026-02-26T03:25:27.436171+0000","last_unstale":"2026-02-26T03:39:45.230701+0000","last_undegraded":"2026-02-26T03:39:45.230701+0000","last_fullsized":"2026-02-26T03:39:45.230701+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1771,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.230657+0000","last_change":"2026-02-26T03:25:29.393046+0000","last_active":"2026-02-26T03:39:45.230657+0000","last_peered":"2026-02-26T03:39:45.230657+0000","last_clean":"2026-02-26T03:39:45.230657+0000","last_became_active":"2026-02-26T03:25:27.412304+0000","last_became_peered":"2026-02-26T03:25:27.412304+0000","last_unstale":"2026-02-26T03:39:45.230657+0000","last_undegraded":"2026-02-26T03:39:45.230657+0000","last_fullsized":"2026-02-26T03:39:45.230657+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:26.360795+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1845,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:45.229641+0000","last_change":"2026-02-26T03:25:24.393577+0000","last_active":"2026-02-26T03:39:45.229641+0000","last_peered":"2026-02-26T03:39:45.229641+0000","last_clean":"2026-02-26T03:39:45.229641+0000","last_became_active":"2026-02-26T03:25:24.384128+0000","last_became_peered":"2026-02-26T03:25:24.384128+0000","last_unstale":"2026-02-26T03:39:45.229641+0000","last_undegraded":"2026-02-26T03:39:45.229641+0000","last_fullsized":"2026-02-26T03:39:45.229641+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:25:23.349596+0000","last_clean_scrub_stamp":"2026-02-26T03:25:23.349596+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160248385536,"internally_reserved":0,"allocated":7581696,"data_stored":7581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53415813120,"internally_reserved":0,"allocated":2842624,"data_stored":2842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1920,"kb_used_data":1920,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164736,"statfs":{"total":53418655744,"available":53416689664,"internally_reserved":0,"allocated":1966080,"data_stored":1966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2708,"kb_used_data":2708,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52163948,"statfs":{"total":53418655744,"available":53415882752,"internally_reserved":0,"allocated":2772992,"data_stored":2772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":3064,"kb_used_data":3064,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52163592,"statfs":{"total":53418655744,"available":53415518208,"internally_reserved":0,"allocated":3137536,"data_stored":3137536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53415813120,"internally_reserved":0,"allocated":2842624,"data_stored":2842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53415882752,"internally_reserved":0,"allocated":2772992,"data_stored":2772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53415813120,"internally_reserved":0,"allocated":2842624,"data_stored":2842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53416689664,"internally_reserved":0,"allocated":1966080,"data_stored":1966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:39:47.836 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.837 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.837 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.837 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.837 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.838 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.838 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.838 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-26T03:25:26.360795+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=26, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.838 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-26T03:25:23.349596+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=25, tm_sec=23, tm_wday=3, tm_yday=57, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=26, tm_hour=3, tm_min=39, tm_sec=45, tm_wday=3, tm_yday=57, tm_isdst=0) 2026-02-26T03:39:47.838 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-26T03:40:07.840 DEBUG:teuthology.orchestra.run.trial181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:40:07.918 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:40:07.914+0000 7f06c8bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:40:07.920 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:40:07.919+0000 7f06c8bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:40:08.033 INFO:teuthology.orchestra.run.trial181.stdout: 2026-02-26T03:40:08.033 INFO:teuthology.orchestra.run.trial181.stderr:dumped all 2026-02-26T03:40:08.042 INFO:teuthology.orchestra.run.trial181.stdout:{"pg_ready":true,"pg_map":{"version":1040,"stamp":"2026-02-26T03:40:06.671443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":7404,"kb_used_data":7404,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156492564,"statfs":{"total":160255967232,"available":160248385536,"internally_reserved":0,"allocated":7581696,"data_stored":7581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001913"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1772,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.325809+0000","last_change":"2026-02-26T03:39:47.340293+0000","last_active":"2026-02-26T03:39:47.325809+0000","last_peered":"2026-02-26T03:39:47.325809+0000","last_clean":"2026-02-26T03:39:47.325809+0000","last_became_active":"2026-02-26T03:25:27.412247+0000","last_became_peered":"2026-02-26T03:25:27.412247+0000","last_unstale":"2026-02-26T03:39:47.325809+0000","last_undegraded":"2026-02-26T03:39:47.325809+0000","last_fullsized":"2026-02-26T03:39:47.325809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:39:47.340241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.340241+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1772,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.325734+0000","last_change":"2026-02-26T03:39:47.336425+0000","last_active":"2026-02-26T03:39:47.325734+0000","last_peered":"2026-02-26T03:39:47.325734+0000","last_clean":"2026-02-26T03:39:47.325734+0000","last_became_active":"2026-02-26T03:25:27.415950+0000","last_became_peered":"2026-02-26T03:25:27.415950+0000","last_unstale":"2026-02-26T03:39:47.325734+0000","last_undegraded":"2026-02-26T03:39:47.325734+0000","last_fullsized":"2026-02-26T03:39:47.325734+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:39:47.336352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.336352+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1772,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.325603+0000","last_change":"2026-02-26T03:39:47.339633+0000","last_active":"2026-02-26T03:39:47.325603+0000","last_peered":"2026-02-26T03:39:47.325603+0000","last_clean":"2026-02-26T03:39:47.325603+0000","last_became_active":"2026-02-26T03:25:27.421547+0000","last_became_peered":"2026-02-26T03:25:27.421547+0000","last_unstale":"2026-02-26T03:39:47.325603+0000","last_undegraded":"2026-02-26T03:39:47.325603+0000","last_fullsized":"2026-02-26T03:39:47.325603+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:39:47.339579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.339579+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1772,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.324940+0000","last_change":"2026-02-26T03:39:47.338402+0000","last_active":"2026-02-26T03:39:47.324940+0000","last_peered":"2026-02-26T03:39:47.324940+0000","last_clean":"2026-02-26T03:39:47.324940+0000","last_became_active":"2026-02-26T03:25:27.419349+0000","last_became_peered":"2026-02-26T03:25:27.419349+0000","last_unstale":"2026-02-26T03:39:47.324940+0000","last_undegraded":"2026-02-26T03:39:47.324940+0000","last_fullsized":"2026-02-26T03:39:47.324940+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:39:47.338339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.338339+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1774,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.025658+0000","last_change":"2026-02-26T03:39:47.035766+0000","last_active":"2026-02-26T03:39:47.025658+0000","last_peered":"2026-02-26T03:39:47.025658+0000","last_clean":"2026-02-26T03:39:47.025658+0000","last_became_active":"2026-02-26T03:25:27.431933+0000","last_became_peered":"2026-02-26T03:25:27.431933+0000","last_unstale":"2026-02-26T03:39:47.025658+0000","last_undegraded":"2026-02-26T03:39:47.025658+0000","last_fullsized":"2026-02-26T03:39:47.025658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'2","last_scrub_stamp":"2026-02-26T03:39:47.035673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.035673+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1772,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.622781+0000","last_change":"2026-02-26T03:39:47.630787+0000","last_active":"2026-02-26T03:39:47.622781+0000","last_peered":"2026-02-26T03:39:47.622781+0000","last_clean":"2026-02-26T03:39:47.622781+0000","last_became_active":"2026-02-26T03:25:27.431293+0000","last_became_peered":"2026-02-26T03:25:27.431293+0000","last_unstale":"2026-02-26T03:39:47.622781+0000","last_undegraded":"2026-02-26T03:39:47.622781+0000","last_fullsized":"2026-02-26T03:39:47.622781+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:39:47.630720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.630720+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1772,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.621665+0000","last_change":"2026-02-26T03:39:47.629550+0000","last_active":"2026-02-26T03:39:47.621665+0000","last_peered":"2026-02-26T03:39:47.621665+0000","last_clean":"2026-02-26T03:39:47.621665+0000","last_became_active":"2026-02-26T03:25:27.436171+0000","last_became_peered":"2026-02-26T03:25:27.436171+0000","last_unstale":"2026-02-26T03:39:47.621665+0000","last_undegraded":"2026-02-26T03:39:47.621665+0000","last_fullsized":"2026-02-26T03:39:47.621665+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:39:47.629485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.629485+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1773,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.325009+0000","last_change":"2026-02-26T03:39:47.337527+0000","last_active":"2026-02-26T03:39:47.325009+0000","last_peered":"2026-02-26T03:39:47.325009+0000","last_clean":"2026-02-26T03:39:47.325009+0000","last_became_active":"2026-02-26T03:25:27.412304+0000","last_became_peered":"2026-02-26T03:25:27.412304+0000","last_unstale":"2026-02-26T03:39:47.325009+0000","last_undegraded":"2026-02-26T03:39:47.325009+0000","last_fullsized":"2026-02-26T03:39:47.325009+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'1","last_scrub_stamp":"2026-02-26T03:39:47.337425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.337425+0000","last_clean_scrub_stamp":"2026-02-26T03:25:26.360795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1847,"reported_epoch":602,"state":"active+clean","last_fresh":"2026-02-26T03:39:47.323715+0000","last_change":"2026-02-26T03:39:47.349761+0000","last_active":"2026-02-26T03:39:47.323715+0000","last_peered":"2026-02-26T03:39:47.323715+0000","last_clean":"2026-02-26T03:39:47.323715+0000","last_became_active":"2026-02-26T03:25:24.384128+0000","last_became_peered":"2026-02-26T03:25:24.384128+0000","last_unstale":"2026-02-26T03:39:47.323715+0000","last_undegraded":"2026-02-26T03:39:47.323715+0000","last_fullsized":"2026-02-26T03:39:47.323715+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2026-02-26T03:39:47.349698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:39:47.349698+0000","last_clean_scrub_stamp":"2026-02-26T03:25:23.349596+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160248385536,"internally_reserved":0,"allocated":7581696,"data_stored":7581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53415813120,"internally_reserved":0,"allocated":2842624,"data_stored":2842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1920,"kb_used_data":1920,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52164736,"statfs":{"total":53418655744,"available":53416689664,"internally_reserved":0,"allocated":1966080,"data_stored":1966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2708,"kb_used_data":2708,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52163948,"statfs":{"total":53418655744,"available":53415882752,"internally_reserved":0,"allocated":2772992,"data_stored":2772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":2776,"kb_used_data":2776,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52163880,"statfs":{"total":53418655744,"available":53415813120,"internally_reserved":0,"allocated":2842624,"data_stored":2842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53415813120,"internally_reserved":0,"allocated":2842624,"data_stored":2842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53415882752,"internally_reserved":0,"allocated":2772992,"data_stored":2772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53415813120,"internally_reserved":0,"allocated":2842624,"data_stored":2842624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53416689664,"internally_reserved":0,"allocated":1966080,"data_stored":1966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:40:08.044 DEBUG:teuthology.orchestra.run.trial181:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:40:08.120 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:40:08.114+0000 7f75cab74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:40:08.122 INFO:teuthology.orchestra.run.trial181.stderr:2026-02-26T03:40:08.120+0000 7f75cab74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:40:08.250 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:40:08.251 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:40:08.251 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:40:08.251 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:40:08.326 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:40:08.326 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:40:08.326 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:40:08.393 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:40:08.394 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:40:08.394 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:40:08.465 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:40:08.465 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:40:08.465 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:40:08.465 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:40:08.499 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:40:08.500 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:40:08.500 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:40:08.500 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:40:08.550 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:40:08.551 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:40:08.551 DEBUG:teuthology.orchestra.run.trial181:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(SLOW_OPS\)' | grep -E -v 'slow request' | head -n 1 2026-02-26T03:40:08.619 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:40:08.619 DEBUG:teuthology.orchestra.run.trial181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:40:08.687 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:40:08.687 DEBUG:teuthology.orchestra.run.trial181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:40:08.752 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:40:08.752 DEBUG:teuthology.orchestra.run.trial181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:40:08.817 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:40:08.818 DEBUG:teuthology.misc:Transferring archived files from trial181:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71183/data/mon.a.tgz 2026-02-26T03:40:08.818 DEBUG:teuthology.orchestra.run.trial181:> mktemp 2026-02-26T03:40:08.872 INFO:teuthology.orchestra.run.trial181.stdout:/tmp/tmp.BAeAtDRgIs 2026-02-26T03:40:08.872 DEBUG:teuthology.orchestra.run.trial181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BAeAtDRgIs 2026-02-26T03:40:08.969 DEBUG:teuthology.orchestra.run.trial181:> sudo chmod 0666 /tmp/tmp.BAeAtDRgIs 2026-02-26T03:40:09.049 DEBUG:teuthology.orchestra.remote:trial181:/tmp/tmp.BAeAtDRgIs is 264KB 2026-02-26T03:40:09.110 DEBUG:teuthology.orchestra.run.trial181:> rm -fr /tmp/tmp.BAeAtDRgIs 2026-02-26T03:40:09.127 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:40:09.128 DEBUG:teuthology.orchestra.run.trial181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:40:09.248 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:40:09.248 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:40:09.250 DEBUG:teuthology.misc:Transferring archived files from trial181:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71183/remote/trial181/crash 2026-02-26T03:40:09.252 DEBUG:teuthology.orchestra.run.trial181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:40:09.311 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:40:09.312 DEBUG:teuthology.orchestra.run.trial181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:40:09.375 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39417.log 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39417.log.gz 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39126.log 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38641.log 2026-02-26T03:40:09.390 INFO:teuthology.orchestra.run.trial181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39126.log.gz 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39274.log 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38641.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.38641.log.gz 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39221.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42029.log 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38731.log 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39274.log.gz 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.42029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37849.log 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42029.log.gz 2026-02-26T03:40:09.391 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42079.log 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38731.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38731.log.gz 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.37849.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37849.log.gz 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39275.log 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.42079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41833.log 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42079.log.gz 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38312.log 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39275.log.gz 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41833.log: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.41833.log.gz 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39128.log 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38312.log.gz 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38780.log 2026-02-26T03:40:09.392 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39128.log.gz 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38878.log 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41588.log 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38780.log.gz 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38980.log 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38878.log.gz 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39368.log 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41588.log.gz 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38980.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38980.log.gz 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38829.log 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38046.log 2026-02-26T03:40:09.393 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39368.log.gz 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38829.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38829.log.gz 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38046.log.gz 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37802.log 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39742.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.36420.log 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.37802.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37802.log.gz 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph.tmp-client.admin.36420.log: 66.4% -- replaced with /var/log/ceph/ceph.tmp-client.admin.36420.log.gz 2026-02-26T03:40:09.394 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41735.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41980.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41735.log: 86.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38666.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41735.log.gz 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41980.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41980.log.gz 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38343.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.38666.log.gz 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38493.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41882.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38343.log.gz 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2026-02-26T03:40:09.395 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38493.log.gz 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41882.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41882.log.gz 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38571.log 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37999.log 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38093.log 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38571.log.gz 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.37999.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37999.log.gz 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38981.log 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41784.log 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38093.log.gz 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.38981.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38981.log.gz 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:40:09.396 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41931.log 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41784.log.gz 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42126.log 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41931.log.gz 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.42126.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42126.log.gz 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39127.log 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.41539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.39127.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39127.log.gz 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr:/var/log/ceph/ceph-client.admin.37706.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2026-02-26T03:40:09.397 INFO:teuthology.orchestra.run.trial181.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:40:09.433 INFO:teuthology.orchestra.run.trial181.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:40:09.635 INFO:teuthology.orchestra.run.trial181.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T03:40:13.725 INFO:teuthology.orchestra.run.trial181.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:40:15.159 INFO:teuthology.orchestra.run.trial181.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:40:20.774 INFO:teuthology.orchestra.run.trial181.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:40:20.776 INFO:teuthology.orchestra.run.trial181.stderr: 2026-02-26T03:40:20.776 INFO:teuthology.orchestra.run.trial181.stderr:real 0m11.409s 2026-02-26T03:40:20.776 INFO:teuthology.orchestra.run.trial181.stderr:user 0m20.526s 2026-02-26T03:40:20.776 INFO:teuthology.orchestra.run.trial181.stderr:sys 0m1.299s 2026-02-26T03:40:20.777 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:40:20.777 DEBUG:teuthology.misc:Transferring archived files from trial181:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71183/remote/trial181/log 2026-02-26T03:40:20.778 DEBUG:teuthology.orchestra.run.trial181:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:40:22.463 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:40:22.532 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:40:22.543 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:40:22.543 DEBUG:teuthology.orchestra.run.trial181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:40:22.568 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:40:22.580 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:40:22.580 DEBUG:teuthology.orchestra.run.trial181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:40:22.624 INFO:teuthology.orchestra.run.trial181.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:40:22.656 INFO:teuthology.orchestra.run.trial181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:40:22.656 INFO:teuthology.orchestra.run.trial181.stdout:=============================================================================== 2026-02-26T03:40:22.656 INFO:teuthology.orchestra.run.trial181.stdout:^- ntp.silvertree.org 1 7 377 83 +6028us[+5917us] +/- 45ms 2026-02-26T03:40:22.657 INFO:teuthology.orchestra.run.trial181.stdout:^- ntp.maxhost.io 2 7 377 8 +299us[ +299us] +/- 55ms 2026-02-26T03:40:22.657 INFO:teuthology.orchestra.run.trial181.stdout:^- ip229.ip-51-81-226.us 2 7 377 16 +1128us[+1128us] +/- 111ms 2026-02-26T03:40:22.657 INFO:teuthology.orchestra.run.trial181.stdout:^* nyc2.us.ntp.li 2 7 377 18 -314us[ -430us] +/- 2872us 2026-02-26T03:40:22.657 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:40:22.669 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:40:22.669 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:40:22.684 DEBUG:teuthology.orchestra.run.trial181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:40:22.752 DEBUG:teuthology.orchestra.run.trial181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:40:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:40:22.780 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:40:22.797 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:40:22.816 INFO:teuthology.task.internal:Duration was 1740.095527 seconds 2026-02-26T03:40:22.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:40:22.838 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:40:22.839 DEBUG:teuthology.orchestra.run.trial181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:40:22.875 INFO:teuthology.orchestra.run.trial181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:40:23.209 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:40:23.210 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial181.front.sepia.ceph.com 2026-02-26T03:40:23.210 DEBUG:teuthology.orchestra.run.trial181:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:40:23.269 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:40:23.270 DEBUG:teuthology.orchestra.run.trial181:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:40:23.371 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:40:23.371 DEBUG:teuthology.orchestra.run.trial181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:40:23.394 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:40:23.395 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:40:23.396 INFO:teuthology.orchestra.run.trial181.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:40:23.396 INFO:teuthology.orchestra.run.trial181.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:40:23.396 INFO:teuthology.orchestra.run.trial181.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:40:23.405 INFO:teuthology.orchestra.run.trial181.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:40:23.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:40:23.425 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:40:23.425 DEBUG:teuthology.orchestra.run.trial181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:40:23.472 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:40:23.489 DEBUG:teuthology.orchestra.run.trial181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:40:23.536 INFO:teuthology.orchestra.run.trial181.stdout:kernel.core_pattern = core 2026-02-26T03:40:23.550 DEBUG:teuthology.orchestra.run.trial181:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:40:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:40:23.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:40:23.624 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:40:23.625 DEBUG:teuthology.misc:Transferring archived files from trial181:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71183/remote/trial181 2026-02-26T03:40:23.625 DEBUG:teuthology.orchestra.run.trial181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:40:23.683 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:40:23.683 DEBUG:teuthology.orchestra.run.trial181:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:40:23.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:40:23.748 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:40:23.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:40:23.764 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:40:23.765 DEBUG:teuthology.orchestra.run.trial181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:40:23.786 INFO:teuthology.orchestra.run.trial181.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:40 /home/ubuntu/cephtest 2026-02-26T03:40:23.787 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:40:23.803 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:40:23.838 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{0-crimson_install clusters/crimson-fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} tasks/rbd_python_api_tests_old_format} duration: 1740.0955274105072 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-26T03:40:23.838 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:40:23.951 INFO:teuthology.run:pass