2026-02-26T03:06:54.469 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T03:06:54.484 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:06:54.566 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71180 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_omap_write} email: null first_in_suite: false flavor: debug job_id: '71180' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5117 sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 targets: trial106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGu4IO27ushH8zqNwnk4x3940jSLIzQAUGkeVqcSExWs3YKuuTDXAKvrhc7NBdf95pbPfC/nb/9Pm9Vfe9zyJuA= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - radosbench: clients: - client.0 objectsize: 4096 size: 4096 time: 300 write-omap: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_02:22:56 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T03:06:54.566 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa; will attempt to use it 2026-02-26T03:06:54.567 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa/tasks 2026-02-26T03:06:54.567 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T03:06:54.569 INFO:teuthology.task.internal:Checking packages... 2026-02-26T03:06:54.569 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'abffb617683951511fccf3c3cab6e219f2ec83a8' 2026-02-26T03:06:54.569 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T03:06:54.569 INFO:teuthology.packaging:ref: None 2026-02-26T03:06:54.569 INFO:teuthology.packaging:tag: None 2026-02-26T03:06:54.569 INFO:teuthology.packaging:branch: main 2026-02-26T03:06:54.569 INFO:teuthology.packaging:sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:06:54.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-26T03:06:54.698 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5593.g63656871 2026-02-26T03:06:54.699 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T03:06:54.710 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T03:06:54.710 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T03:06:54.724 INFO:teuthology.task.internal:Saving configuration 2026-02-26T03:06:54.735 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T03:06:54.747 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T03:06:54.816 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial106.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71180', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 03:03:23.975053', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGu4IO27ushH8zqNwnk4x3940jSLIzQAUGkeVqcSExWs3YKuuTDXAKvrhc7NBdf95pbPfC/nb/9Pm9Vfe9zyJuA='} 2026-02-26T03:06:54.816 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T03:06:54.828 INFO:teuthology.task.internal:roles: ubuntu@trial106.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T03:06:54.828 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T03:06:54.917 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x730731a60220>, signals=[15]) 2026-02-26T03:06:54.917 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T03:06:54.929 INFO:teuthology.task.internal:Opening connections... 2026-02-26T03:06:54.929 DEBUG:teuthology.task.internal:connecting to ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:06:54.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:06:55.009 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T03:06:55.020 DEBUG:teuthology.orchestra.run.trial106:> uname -m 2026-02-26T03:06:55.083 INFO:teuthology.orchestra.run.trial106.stdout:x86_64 2026-02-26T03:06:55.084 DEBUG:teuthology.orchestra.run.trial106:> cat /etc/os-release 2026-02-26T03:06:55.139 INFO:teuthology.orchestra.run.trial106.stdout:NAME="CentOS Stream" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:VERSION="9" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:ID="centos" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:ID_LIKE="rhel fedora" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:VERSION_ID="9" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:PLATFORM_ID="platform:el9" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:ANSI_COLOR="0;31" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:LOGO="fedora-logo-icon" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:HOME_URL="https://centos.org/" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T03:06:55.140 INFO:teuthology.orchestra.run.trial106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T03:06:55.141 INFO:teuthology.lock.ops:Updating trial106.front.sepia.ceph.com on lock server 2026-02-26T03:06:55.229 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T03:06:55.241 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T03:06:55.252 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T03:06:55.252 DEBUG:teuthology.orchestra.run.trial106:> test '!' -e /home/ubuntu/cephtest 2026-02-26T03:06:55.268 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T03:06:55.280 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T03:06:55.280 DEBUG:teuthology.orchestra.run.trial106:> test -z $(ls -A /var/lib/ceph) 2026-02-26T03:06:55.324 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T03:06:55.403 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T03:06:55.424 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T03:06:55.424 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T03:06:55.424 DEBUG:teuthology.orchestra.run.trial106:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T03:06:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:06:55.439 DEBUG:teuthology.orchestra.run.trial106:> uname -r 2026-02-26T03:06:55.495 INFO:teuthology.orchestra.run.trial106.stdout:5.14.0-681.el9.x86_64 2026-02-26T03:06:55.495 INFO:teuthology.task.kernel:Running kernel on trial106: 5.14.0-681.el9.x86_64 2026-02-26T03:06:55.495 DEBUG:teuthology.orchestra.run.trial106:> sudo yum install -y kernel 2026-02-26T03:06:57.255 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.9 MB 00:01 2026-02-26T03:07:24.800 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 27 MB 00:27 2026-02-26T03:07:36.318 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - CRB 890 kB/s | 8.0 MB 00:09 2026-02-26T03:07:37.254 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - Extras packages 105 kB/s | 20 kB 00:00 2026-02-26T03:07:37.852 INFO:teuthology.orchestra.run.trial106.stdout:Extra Packages for Enterprise Linux 38 MB/s | 20 MB 00:00 2026-02-26T03:07:40.988 INFO:teuthology.orchestra.run.trial106.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-26T03:07:42.014 INFO:teuthology.orchestra.run.trial106.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T03:07:42.014 INFO:teuthology.orchestra.run.trial106.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T03:07:42.034 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout: Package Architecture Version Repository Size 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout:Installing dependencies: 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T03:07:42.035 INFO:teuthology.orchestra.run.trial106.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T03:07:42.036 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:07:42.036 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:07:42.036 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:07:42.036 INFO:teuthology.orchestra.run.trial106.stdout:Install 4 Packages 2026-02-26T03:07:42.036 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:07:42.036 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 89 M 2026-02-26T03:07:42.036 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 135 M 2026-02-26T03:07:42.037 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:07:42.402 INFO:teuthology.orchestra.run.trial106.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 14 MB/s | 901 kB 00:00 2026-02-26T03:07:42.669 INFO:teuthology.orchestra.run.trial106.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 52 MB/s | 17 MB 00:00 2026-02-26T03:07:43.047 INFO:teuthology.orchestra.run.trial106.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 56 MB/s | 40 MB 00:00 2026-02-26T03:07:43.273 INFO:teuthology.orchestra.run.trial106.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 36 MB/s | 31 MB 00:00 2026-02-26T03:07:43.273 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:07:43.273 INFO:teuthology.orchestra.run.trial106.stdout:Total 72 MB/s | 89 MB 00:01 2026-02-26T03:07:43.465 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:07:43.667 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:07:43.668 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:07:44.050 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:07:44.050 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:07:45.156 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:07:45.444 INFO:teuthology.orchestra.run.trial106.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:07:45.466 INFO:teuthology.orchestra.run.trial106.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:07:48.026 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:07:48.059 INFO:teuthology.orchestra.run.trial106.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:07:51.098 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:07:51.104 INFO:teuthology.orchestra.run.trial106.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:07:54.217 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:08:11.795 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:08:11.816 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:08:12.004 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:08:12.004 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:08:12.004 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:08:12.005 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:08:12.063 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:08:12.123 DEBUG:teuthology.orchestra.run.trial106:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:08:12.154 INFO:teuthology.orchestra.run.trial106.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.154 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:08:12.154 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T03:08:12.154 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T03:08:12.155 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T03:08:12.155 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T03:08:12.155 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial106.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T03:08:12.155 DEBUG:teuthology.orchestra.run.trial106:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:08:12.219 INFO:teuthology.orchestra.run.trial106.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.221 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:08:12.221 DEBUG:teuthology.orchestra.run.trial106:> sudo rpm -qi grub2-tools 2026-02-26T03:08:12.290 INFO:teuthology.orchestra.run.trial106.stdout:Name : grub2-tools 2026-02-26T03:08:12.290 INFO:teuthology.orchestra.run.trial106.stdout:Epoch : 1 2026-02-26T03:08:12.290 INFO:teuthology.orchestra.run.trial106.stdout:Version : 2.06 2026-02-26T03:08:12.290 INFO:teuthology.orchestra.run.trial106.stdout:Release : 120.el9 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Architecture: x86_64 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Group : Unspecified 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Size : 9108611 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:License : GPLv3+ 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Packager : builder@centos.org 2026-02-26T03:08:12.291 INFO:teuthology.orchestra.run.trial106.stdout:Vendor : CentOS 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:Summary : Support tools for GRUB. 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:Description : 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:hardware devices. 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:08:12.292 INFO:teuthology.orchestra.run.trial106.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T03:08:12.293 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T03:08:12.294 INFO:teuthology.task.kernel:Updating grub on trial106 to boot 5.14.0-686.el9.x86_64 2026-02-26T03:08:12.294 DEBUG:teuthology.orchestra.run.trial106:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T03:08:12.357 INFO:teuthology.orchestra.run.trial106.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T03:08:12.359 DEBUG:teuthology.orchestra.run.trial106:> sudo command -v grubby && echo yes || echo no 2026-02-26T03:08:12.423 INFO:teuthology.orchestra.run.trial106.stdout:/sbin/grubby 2026-02-26T03:08:12.424 INFO:teuthology.orchestra.run.trial106.stdout:yes 2026-02-26T03:08:12.425 DEBUG:teuthology.orchestra.run.trial106:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T03:08:12.489 INFO:teuthology.orchestra.run.trial106.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T03:08:12.489 DEBUG:teuthology.orchestra.run.trial106:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.562 DEBUG:teuthology.orchestra.run.trial106:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T03:08:12.773 INFO:teuthology.orchestra.run.trial106.stderr:Generating grub configuration file ... 2026-02-26T03:08:12.840 INFO:teuthology.orchestra.run.trial106.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T03:08:12.847 INFO:teuthology.orchestra.run.trial106.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T03:08:13.097 INFO:teuthology.orchestra.run.trial106.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T03:08:13.104 INFO:teuthology.orchestra.run.trial106.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T03:08:13.137 INFO:teuthology.orchestra.run.trial106.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T03:08:13.144 INFO:teuthology.orchestra.run.trial106.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T03:08:13.175 INFO:teuthology.orchestra.run.trial106.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T03:08:13.182 INFO:teuthology.orchestra.run.trial106.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T03:08:13.207 INFO:teuthology.orchestra.run.trial106.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T03:08:13.213 INFO:teuthology.orchestra.run.trial106.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T03:08:13.301 INFO:teuthology.orchestra.run.trial106.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T03:08:13.310 INFO:teuthology.orchestra.run.trial106.stderr:done 2026-02-26T03:08:13.312 DEBUG:teuthology.orchestra.run.trial106:> test -d /sys/firmware/efi 2026-02-26T03:08:13.328 DEBUG:teuthology.orchestra.run.trial106:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T03:08:13.394 INFO:teuthology.orchestra.run.trial106.stdout:/boot/efi/EFI/BOOT 2026-02-26T03:08:13.394 INFO:teuthology.orchestra.run.trial106.stdout:/boot/efi/EFI/centos 2026-02-26T03:08:13.396 DEBUG:teuthology.orchestra.run.trial106:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T03:08:13.463 INFO:teuthology.orchestra.run.trial106.stdout:no 2026-02-26T03:08:13.463 DEBUG:teuthology.orchestra.run.trial106:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T03:08:13.527 INFO:teuthology.orchestra.run.trial106.stdout:yes 2026-02-26T03:08:13.528 DEBUG:teuthology.orchestra.run.trial106:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T03:08:13.592 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial106 2026-02-26T03:08:13.592 DEBUG:teuthology.orchestra.run.trial106:> sudo bash -c ' 2026-02-26T03:08:13.592 DEBUG:teuthology.orchestra.run.trial106:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T03:08:13.592 DEBUG:teuthology.orchestra.run.trial106:> ' 2026-02-26T03:08:43.636 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T03:08:43.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial106.front.sepia.ceph.com' 2026-02-26T03:08:43.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:09:43.686 DEBUG:teuthology.orchestra.remote:timed out 2026-02-26T03:09:52.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial106.front.sepia.ceph.com' 2026-02-26T03:09:52.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:09:52.824 DEBUG:teuthology.orchestra.run.trial106:> true 2026-02-26T03:09:52.979 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial106.front.sepia.ceph.com' 2026-02-26T03:09:52.980 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T03:09:52.980 DEBUG:teuthology.orchestra.run.trial106:> uname -r 2026-02-26T03:09:53.037 INFO:teuthology.orchestra.run.trial106.stdout:5.14.0-686.el9.x86_64 2026-02-26T03:09:53.037 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T03:09:53.038 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T03:09:53.038 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T03:09:54.038 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T03:09:54.039 DEBUG:teuthology.orchestra.run.trial106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T03:09:54.067 INFO:teuthology.orchestra.run.trial106.stdout:ttyS1 2026-02-26T03:09:54.096 DEBUG:teuthology.parallel:result is None 2026-02-26T03:09:54.096 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T03:09:54.106 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T03:09:54.106 DEBUG:teuthology.orchestra.run.trial106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T03:09:54.152 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T03:09:54.164 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T03:09:54.177 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T03:09:54.177 DEBUG:teuthology.orchestra.run.trial106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T03:09:54.215 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T03:09:54.227 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T03:09:54.227 DEBUG:teuthology.orchestra.run.trial106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T03:09:54.274 INFO:teuthology.orchestra.run.trial106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:09:54.283 INFO:teuthology.orchestra.run.trial106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:09:54.284 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T03:09:54.295 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T03:09:54.295 DEBUG:teuthology.orchestra.run.trial106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T03:09:54.348 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T03:09:54.362 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T03:09:54.362 DEBUG:teuthology.orchestra.run.trial106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T03:09:54.405 DEBUG:teuthology.orchestra.run.trial106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:09:54.548 DEBUG:teuthology.orchestra.run.trial106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:09:54.573 DEBUG:teuthology.orchestra.run.trial106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:09:54.709 DEBUG:teuthology.orchestra.run.trial106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:09:54.735 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:09:54.735 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T03:09:54.800 DEBUG:teuthology.orchestra.run.trial106:> sudo service rsyslog restart 2026-02-26T03:09:54.868 INFO:teuthology.orchestra.run.trial106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:09:55.268 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T03:09:55.286 INFO:teuthology.task.internal:Starting timer... 2026-02-26T03:09:55.286 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T03:09:55.310 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T03:09:55.328 DEBUG:teuthology.orchestra.run.trial106:> sudo service auditd rotate 2026-02-26T03:09:55.356 INFO:teuthology.orchestra.run.trial106.stdout:Rotating logs: 2026-02-26T03:09:55.358 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T03:09:55.358 DEBUG:teuthology.orchestra.run.trial106:> /usr/sbin/getenforce 2026-02-26T03:09:55.414 INFO:teuthology.orchestra.run.trial106.stdout:Permissive 2026-02-26T03:09:55.415 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial106.front.sepia.ceph.com': 'permissive'} 2026-02-26T03:09:55.415 DEBUG:teuthology.orchestra.run.trial106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:09:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:09:55.479 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T03:09:55.480 DEBUG:teuthology.orchestra.run.trial106:> sudo /usr/sbin/setenforce permissive 2026-02-26T03:09:55.542 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T03:09:55.555 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-26T03:09:55.558 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-26T03:09:55.664 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T03:09:55.677 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T03:09:55.679 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial106.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T03:12:41.513 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial106.front.sepia.ceph.com')] 2026-02-26T03:12:41.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial106.front.sepia.ceph.com' 2026-02-26T03:12:41.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:12:41.630 DEBUG:teuthology.orchestra.run.trial106:> true 2026-02-26T03:12:41.701 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial106.front.sepia.ceph.com' 2026-02-26T03:12:41.701 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T03:12:41.715 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T03:12:41.715 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T03:12:41.715 DEBUG:teuthology.orchestra.run.trial106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:12:41.768 INFO:teuthology.orchestra.run.trial106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T03:12:41.780 INFO:teuthology.orchestra.run.trial106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T03:12:41.800 INFO:teuthology.orchestra.run.trial106.stderr:sudo: ntpd: command not found 2026-02-26T03:12:41.810 INFO:teuthology.orchestra.run.trial106.stdout:506 Cannot talk to daemon 2026-02-26T03:12:41.821 INFO:teuthology.orchestra.run.trial106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T03:12:41.833 INFO:teuthology.orchestra.run.trial106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T03:12:41.878 INFO:teuthology.orchestra.run.trial106.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:12:41.897 INFO:teuthology.orchestra.run.trial106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:12:41.897 INFO:teuthology.orchestra.run.trial106.stdout:=============================================================================== 2026-02-26T03:12:41.897 INFO:teuthology.orchestra.run.trial106.stdout:^? static.36.62.78.5.client> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:12:41.897 INFO:teuthology.orchestra.run.trial106.stdout:^? t1.time.bf1.yahoo.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:12:41.897 INFO:teuthology.orchestra.run.trial106.stdout:^? 198-12-95-197-host.coloc> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:12:41.897 INFO:teuthology.orchestra.run.trial106.stdout:^? 99-28-14-242.lightspeed.> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:12:41.899 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:12:41.910 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:12:41.911 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:12:41.911 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:12:41.911 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:12:41.914 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:12:41.914 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-26T03:12:41.914 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:12:41.914 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:12:42.074 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:12:42.074 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:12:42.399 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:12:42.400 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:12:42.400 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:12:42.427 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-26T03:12:42.428 DEBUG:teuthology.orchestra.run.trial106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:12:42.493 DEBUG:teuthology.orchestra.run.trial106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:12:42.565 DEBUG:teuthology.orchestra.run.trial106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:12:42.628 INFO:teuthology.orchestra.run.trial106.stdout:check_obsoletes = 1 2026-02-26T03:12:42.629 DEBUG:teuthology.orchestra.run.trial106:> sudo yum clean all 2026-02-26T03:12:42.780 INFO:teuthology.orchestra.run.trial106.stdout:41 files removed 2026-02-26T03:12:42.795 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-radosgw 2026-02-26T03:12:43.635 INFO:teuthology.orchestra.run.trial106.stdout:ceph packages for x86_64 136 kB/s | 94 kB 00:00 2026-02-26T03:12:44.280 INFO:teuthology.orchestra.run.trial106.stdout:ceph noarch packages 40 kB/s | 24 kB 00:00 2026-02-26T03:12:44.849 INFO:teuthology.orchestra.run.trial106.stdout:ceph source packages 3.8 kB/s | 2.1 kB 00:00 2026-02-26T03:12:46.618 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.9 MB 00:01 2026-02-26T03:12:57.263 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 27 MB 00:10 2026-02-26T03:13:00.677 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 8.0 MB 00:01 2026-02-26T03:13:01.655 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 20 kB 00:00 2026-02-26T03:13:03.379 INFO:teuthology.orchestra.run.trial106.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2026-02-26T03:13:06.548 INFO:teuthology.orchestra.run.trial106.stdout:lab-extras 1.7 MB/s | 50 kB 00:00 2026-02-26T03:13:07.774 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:13:07.776 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:13:07.776 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repository Size 2026-02-26T03:13:07.776 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:13:07.777 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:13:07.777 INFO:teuthology.orchestra.run.trial106.stdout: ceph-radosgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 11 M 2026-02-26T03:13:07.777 INFO:teuthology.orchestra.run.trial106.stdout:Upgrading: 2026-02-26T03:13:07.777 INFO:teuthology.orchestra.run.trial106.stdout: librados2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 3.3 M 2026-02-26T03:13:07.777 INFO:teuthology.orchestra.run.trial106.stdout: librbd1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.7 M 2026-02-26T03:13:07.777 INFO:teuthology.orchestra.run.trial106.stdout:Installing dependencies: 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: ceph-base x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.1 M 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 21 M 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: ceph-selinux x86_64 2:20.3.0-5312.gabffb617.el9 ceph 25 k 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 859 k 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T03:13:07.778 INFO:teuthology.orchestra.run.trial106.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: libradosstriper1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 263 k 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: librgw2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.6 M 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T03:13:07.779 INFO:teuthology.orchestra.run.trial106.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T03:13:07.780 INFO:teuthology.orchestra.run.trial106.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T03:13:07.780 INFO:teuthology.orchestra.run.trial106.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T03:13:07.780 INFO:teuthology.orchestra.run.trial106.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 45 k 2026-02-26T03:13:07.780 INFO:teuthology.orchestra.run.trial106.stdout: python3-ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 185 k 2026-02-26T03:13:07.780 INFO:teuthology.orchestra.run.trial106.stdout: python3-cephfs x86_64 2:20.3.0-5312.gabffb617.el9 ceph 203 k 2026-02-26T03:13:07.780 INFO:teuthology.orchestra.run.trial106.stdout: python3-rados x86_64 2:20.3.0-5312.gabffb617.el9 ceph 324 k 2026-02-26T03:13:07.780 INFO:teuthology.orchestra.run.trial106.stdout: python3-rbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 305 k 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: python3-rgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 100 k 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout:Installing weak dependencies: 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 26 k 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T03:13:07.781 INFO:teuthology.orchestra.run.trial106.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T03:13:07.782 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:07.782 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:13:07.782 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:13:07.782 INFO:teuthology.orchestra.run.trial106.stdout:Install 35 Packages 2026-02-26T03:13:07.782 INFO:teuthology.orchestra.run.trial106.stdout:Upgrade 2 Packages 2026-02-26T03:13:07.782 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:07.783 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 61 M 2026-02-26T03:13:07.783 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:13:15.649 INFO:teuthology.orchestra.run.trial106.stdout:(1/37): ceph-base-20.3.0-5312.gabffb617.el9.x86 710 kB/s | 5.1 MB 00:07 2026-02-26T03:13:15.719 INFO:teuthology.orchestra.run.trial106.stdout:(2/37): ceph-selinux-20.3.0-5312.gabffb617.el9. 360 kB/s | 25 kB 00:00 2026-02-26T03:13:15.790 INFO:teuthology.orchestra.run.trial106.stdout:(3/37): libcephfs-daemon-20.3.0-5312.gabffb617. 524 kB/s | 37 kB 00:00 2026-02-26T03:13:15.861 INFO:teuthology.orchestra.run.trial106.stdout:(4/37): libcephfs-proxy2-20.3.0-5312.gabffb617. 359 kB/s | 26 kB 00:00 2026-02-26T03:13:16.877 INFO:teuthology.orchestra.run.trial106.stdout:(5/37): libcephfs2-20.3.0-5312.gabffb617.el9.x8 846 kB/s | 859 kB 00:01 2026-02-26T03:13:17.404 INFO:teuthology.orchestra.run.trial106.stdout:(6/37): libradosstriper1-20.3.0-5312.gabffb617. 500 kB/s | 263 kB 00:00 2026-02-26T03:13:24.084 INFO:teuthology.orchestra.run.trial106.stdout:(7/37): librgw2-20.3.0-5312.gabffb617.el9.x86_6 862 kB/s | 5.6 MB 00:06 2026-02-26T03:13:24.152 INFO:teuthology.orchestra.run.trial106.stdout:(8/37): python3-ceph-argparse-20.3.0-5312.gabff 670 kB/s | 45 kB 00:00 2026-02-26T03:13:24.353 INFO:teuthology.orchestra.run.trial106.stdout:(9/37): python3-ceph-common-20.3.0-5312.gabffb6 918 kB/s | 185 kB 00:00 2026-02-26T03:13:24.562 INFO:teuthology.orchestra.run.trial106.stdout:(10/37): python3-cephfs-20.3.0-5312.gabffb617.e 972 kB/s | 203 kB 00:00 2026-02-26T03:13:24.918 INFO:teuthology.orchestra.run.trial106.stdout:(11/37): python3-rados-20.3.0-5312.gabffb617.el 911 kB/s | 324 kB 00:00 2026-02-26T03:13:25.234 INFO:teuthology.orchestra.run.trial106.stdout:(12/37): ceph-radosgw-20.3.0-5312.gabffb617.el9 675 kB/s | 11 MB 00:16 2026-02-26T03:13:25.266 INFO:teuthology.orchestra.run.trial106.stdout:(13/37): python3-rbd-20.3.0-5312.gabffb617.el9. 878 kB/s | 305 kB 00:00 2026-02-26T03:13:25.278 INFO:teuthology.orchestra.run.trial106.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 2.7 MB/s | 33 kB 00:00 2026-02-26T03:13:25.300 INFO:teuthology.orchestra.run.trial106.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 51 MB/s | 1.2 MB 00:00 2026-02-26T03:13:25.307 INFO:teuthology.orchestra.run.trial106.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 14 MB/s | 93 kB 00:00 2026-02-26T03:13:25.385 INFO:teuthology.orchestra.run.trial106.stdout:(17/37): python3-rgw-20.3.0-5312.gabffb617.el9. 664 kB/s | 100 kB 00:00 2026-02-26T03:13:26.024 INFO:teuthology.orchestra.run.trial106.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 145 kB/s | 104 kB 00:00 2026-02-26T03:13:26.261 INFO:teuthology.orchestra.run.trial106.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 678 kB/s | 160 kB 00:00 2026-02-26T03:13:26.419 INFO:teuthology.orchestra.run.trial106.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 287 kB/s | 45 kB 00:00 2026-02-26T03:13:26.493 INFO:teuthology.orchestra.run.trial106.stdout:(21/37): libnbd-1.20.3-4.el9.x86_64.rpm 148 kB/s | 164 kB 00:01 2026-02-26T03:13:26.657 INFO:teuthology.orchestra.run.trial106.stdout:(22/37): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2026-02-26T03:13:26.816 INFO:teuthology.orchestra.run.trial106.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 1.5 MB/s | 240 kB 00:00 2026-02-26T03:13:26.974 INFO:teuthology.orchestra.run.trial106.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 235 kB/s | 37 kB 00:00 2026-02-26T03:13:27.131 INFO:teuthology.orchestra.run.trial106.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 421 kB/s | 66 kB 00:00 2026-02-26T03:13:27.173 INFO:teuthology.orchestra.run.trial106.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 7.3 MB/s | 308 kB 00:00 2026-02-26T03:13:27.194 INFO:teuthology.orchestra.run.trial106.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 210 MB/s | 4.4 MB 00:00 2026-02-26T03:13:27.197 INFO:teuthology.orchestra.run.trial106.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 7.4 MB/s | 25 kB 00:00 2026-02-26T03:13:27.201 INFO:teuthology.orchestra.run.trial106.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 14 MB/s | 49 kB 00:00 2026-02-26T03:13:27.204 INFO:teuthology.orchestra.run.trial106.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 20 MB/s | 67 kB 00:00 2026-02-26T03:13:27.209 INFO:teuthology.orchestra.run.trial106.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 146 MB/s | 838 kB 00:00 2026-02-26T03:13:27.213 INFO:teuthology.orchestra.run.trial106.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 48 MB/s | 191 kB 00:00 2026-02-26T03:13:27.221 INFO:teuthology.orchestra.run.trial106.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 208 MB/s | 1.6 MB 00:00 2026-02-26T03:13:27.437 INFO:teuthology.orchestra.run.trial106.stdout:(34/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 310 kB/s | 292 kB 00:00 2026-02-26T03:13:30.398 INFO:teuthology.orchestra.run.trial106.stdout:(35/37): librados2-20.3.0-5312.gabffb617.el9.x8 1.0 MB/s | 3.3 MB 00:03 2026-02-26T03:13:31.510 INFO:teuthology.orchestra.run.trial106.stdout:(36/37): librbd1-20.3.0-5312.gabffb617.el9.x86_ 676 kB/s | 2.7 MB 00:04 2026-02-26T03:13:33.192 INFO:teuthology.orchestra.run.trial106.stdout:(37/37): ceph-common-20.3.0-5312.gabffb617.el9. 855 kB/s | 21 MB 00:24 2026-02-26T03:13:33.195 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:13:33.195 INFO:teuthology.orchestra.run.trial106.stdout:Total 2.4 MB/s | 61 MB 00:25 2026-02-26T03:13:33.289 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:13:33.309 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:13:33.309 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:13:33.558 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:13:33.558 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:13:33.861 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:13:34.000 INFO:teuthology.orchestra.run.trial106.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-26T03:13:34.004 INFO:teuthology.orchestra.run.trial106.stdout: Upgrading : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:13:34.052 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:13:34.053 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:13:34.065 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:13:34.067 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 4/39 2026-02-26T03:13:34.068 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:13:34.094 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:13:34.098 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 6/39 2026-02-26T03:13:34.107 INFO:teuthology.orchestra.run.trial106.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-26T03:13:34.109 INFO:teuthology.orchestra.run.trial106.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-26T03:13:34.112 INFO:teuthology.orchestra.run.trial106.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-26T03:13:34.116 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-26T03:13:34.127 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 11/39 2026-02-26T03:13:34.158 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:13:34.159 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:13:34.180 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:13:34.185 INFO:teuthology.orchestra.run.trial106.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-26T03:13:34.215 INFO:teuthology.orchestra.run.trial106.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-26T03:13:34.219 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-26T03:13:34.226 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-26T03:13:34.228 INFO:teuthology.orchestra.run.trial106.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-26T03:13:34.232 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-26T03:13:34.347 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:13:34.348 INFO:teuthology.orchestra.run.trial106.stdout: Upgrading : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:13:34.376 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:13:34.379 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 22/39 2026-02-26T03:13:34.385 INFO:teuthology.orchestra.run.trial106.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-26T03:13:34.633 INFO:teuthology.orchestra.run.trial106.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-26T03:13:34.634 INFO:teuthology.orchestra.run.trial106.stdout: Installing : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:13:34.651 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:13:34.655 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 26/39 2026-02-26T03:13:34.659 INFO:teuthology.orchestra.run.trial106.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-26T03:13:34.661 INFO:teuthology.orchestra.run.trial106.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-26T03:13:34.662 INFO:teuthology.orchestra.run.trial106.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:13:34.680 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:13:34.795 INFO:teuthology.orchestra.run.trial106.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-26T03:13:34.797 INFO:teuthology.orchestra.run.trial106.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:13:35.085 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:13:35.085 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T03:13:35.085 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:35.130 INFO:teuthology.orchestra.run.trial106.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-26T03:13:35.132 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 33/39 2026-02-26T03:13:36.084 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:13:36.086 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:13:36.348 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:13:36.351 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:13:36.392 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:13:36.392 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:13:36.392 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:13:36.392 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:36.395 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:13:40.869 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:13:40.869 INFO:teuthology.orchestra.run.trial106.stdout:skipping the directory /sys 2026-02-26T03:13:40.869 INFO:teuthology.orchestra.run.trial106.stdout:skipping the directory /proc 2026-02-26T03:13:40.869 INFO:teuthology.orchestra.run.trial106.stdout:skipping the directory /mnt 2026-02-26T03:13:40.870 INFO:teuthology.orchestra.run.trial106.stdout:skipping the directory /var/tmp 2026-02-26T03:13:40.870 INFO:teuthology.orchestra.run.trial106.stdout:skipping the directory /home 2026-02-26T03:13:40.870 INFO:teuthology.orchestra.run.trial106.stdout:skipping the directory /root 2026-02-26T03:13:40.870 INFO:teuthology.orchestra.run.trial106.stdout:skipping the directory /tmp 2026-02-26T03:13:40.870 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:41.360 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:13:41.383 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:13:41.384 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:13:41.384 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:13:41.384 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:13:41.384 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:13:41.384 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:41.386 INFO:teuthology.orchestra.run.trial106.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:13:41.399 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:13:41.399 INFO:teuthology.orchestra.run.trial106.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:13:41.456 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:13:41.456 INFO:teuthology.orchestra.run.trial106.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:13:41.456 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:41.761 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 1/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 4/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 6/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 7/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 8/39 2026-02-26T03:13:41.762 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 9/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 10/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 11/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 13/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 14/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 15/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-26T03:13:41.763 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-26T03:13:41.764 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-26T03:13:41.765 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-26T03:13:41.765 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-26T03:13:41.765 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-26T03:13:41.765 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-26T03:13:41.765 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:13:41.765 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-26T03:13:41.765 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 38/39 2026-02-26T03:13:41.826 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:13:41.826 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout:Upgraded: 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.827 INFO:teuthology.orchestra.run.trial106.stdout: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:13:41.828 INFO:teuthology.orchestra.run.trial106.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:13:41.829 INFO:teuthology.orchestra.run.trial106.stdout: python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:13:41.830 INFO:teuthology.orchestra.run.trial106.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:13:41.831 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:41.831 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:13:41.866 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-test 2026-02-26T03:13:42.182 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:36 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repository Size 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: ceph-test x86_64 2:20.3.0-5312.gabffb617.el9 ceph 126 M 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout:Installing dependencies: 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: libcephsqlite x86_64 2:20.3.0-5312.gabffb617.el9 ceph 153 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:13:42.266 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:13:42.267 INFO:teuthology.orchestra.run.trial106.stdout:Install 12 Packages 2026-02-26T03:13:42.267 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:13:42.267 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 131 M 2026-02-26T03:13:42.267 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 976 M 2026-02-26T03:13:42.267 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:13:42.683 INFO:teuthology.orchestra.run.trial106.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2026-02-26T03:13:42.703 INFO:teuthology.orchestra.run.trial106.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.1 MB/s | 23 kB 00:00 2026-02-26T03:13:42.724 INFO:teuthology.orchestra.run.trial106.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.4 MB/s | 94 kB 00:00 2026-02-26T03:13:42.906 INFO:teuthology.orchestra.run.trial106.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 2.8 MB/s | 525 kB 00:00 2026-02-26T03:13:42.936 INFO:teuthology.orchestra.run.trial106.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 5.9 MB/s | 176 kB 00:00 2026-02-26T03:13:42.966 INFO:teuthology.orchestra.run.trial106.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 7.3 MB/s | 233 kB 00:00 2026-02-26T03:13:43.025 INFO:teuthology.orchestra.run.trial106.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2026-02-26T03:13:43.044 INFO:teuthology.orchestra.run.trial106.stdout:(8/12): libcephsqlite-20.3.0-5312.gabffb617.el9 278 kB/s | 153 kB 00:00 2026-02-26T03:13:43.052 INFO:teuthology.orchestra.run.trial106.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 11 MB/s | 303 kB 00:00 2026-02-26T03:13:43.152 INFO:teuthology.orchestra.run.trial106.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 596 kB/s | 64 kB 00:00 2026-02-26T03:13:43.259 INFO:teuthology.orchestra.run.trial106.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 588 kB/s | 122 kB 00:00 2026-02-26T03:15:39.823 INFO:teuthology.orchestra.run.trial106.stdout:(12/12): ceph-test-20.3.0-5312.gabffb617.el9.x8 1.1 MB/s | 126 MB 01:57 2026-02-26T03:15:39.824 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:15:39.825 INFO:teuthology.orchestra.run.trial106.stdout:Total 1.1 MB/s | 131 MB 01:57 2026-02-26T03:15:39.895 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:15:39.905 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:15:39.905 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:15:40.335 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:15:40.336 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:15:40.794 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:15:40.803 INFO:teuthology.orchestra.run.trial106.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-26T03:15:40.812 INFO:teuthology.orchestra.run.trial106.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-26T03:15:40.817 INFO:teuthology.orchestra.run.trial106.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-26T03:15:40.821 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-26T03:15:40.823 INFO:teuthology.orchestra.run.trial106.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-26T03:15:40.828 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-26T03:15:40.829 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:15:40.845 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:15:40.847 INFO:teuthology.orchestra.run.trial106.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-26T03:15:40.859 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-26T03:15:40.867 INFO:teuthology.orchestra.run.trial106.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-26T03:15:40.868 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:15:47.885 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:15:47.904 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:15:48.686 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:15:48.686 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 1/12 2026-02-26T03:15:48.686 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2/12 2026-02-26T03:15:48.686 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-26T03:15:48.686 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-26T03:15:48.687 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-26T03:15:48.687 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-26T03:15:48.687 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-26T03:15:48.687 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-26T03:15:48.687 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-26T03:15:48.687 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-26T03:15:48.687 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:15:48.747 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:15:48.787 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph 2026-02-26T03:15:49.101 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:02:43 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:15:49.187 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout:======================================================================================= 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repository Size 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout:======================================================================================= 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph x86_64 2:20.3.0-5312.gabffb617.el9 ceph 6.5 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout:Installing dependencies: 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 57 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mds x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.0 M 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr x86_64 2:20.3.0-5312.gabffb617.el9 ceph 937 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 297 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 4.9 M 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 163 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.gabffb617.el9 ceph 15 M 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 M 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 17 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 1.0 M 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:15:49.190 INFO:teuthology.orchestra.run.trial106.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:15:49.191 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout:Installing weak dependencies: 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 184 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 51 M 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 7.4 M 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 22 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 50 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: ceph-volume noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 301 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout:======================================================================================= 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout:Install 96 Packages 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:15:49.192 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 225 M 2026-02-26T03:15:49.193 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 1.5 G 2026-02-26T03:15:49.193 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:15:49.777 INFO:teuthology.orchestra.run.trial106.stdout:(1/96): ceph-20.3.0-5312.gabffb617.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2026-02-26T03:15:50.402 INFO:teuthology.orchestra.run.trial106.stdout:(2/96): ceph-mds-20.3.0-5312.gabffb617.el9.x86_ 2.2 MB/s | 2.0 MB 00:00 2026-02-26T03:15:50.470 INFO:teuthology.orchestra.run.trial106.stdout:(3/96): ceph-osd-20.3.0-5312.gabffb617.el9.x86_ 2.3 MB/s | 163 kB 00:00 2026-02-26T03:15:51.122 INFO:teuthology.orchestra.run.trial106.stdout:(4/96): ceph-mgr-20.3.0-5312.gabffb617.el9.x86_ 572 kB/s | 937 kB 00:01 2026-02-26T03:15:58.290 INFO:teuthology.orchestra.run.trial106.stdout:(5/96): ceph-osd-classic-20.3.0-5312.gabffb617. 1.9 MB/s | 15 MB 00:07 2026-02-26T03:15:58.359 INFO:teuthology.orchestra.run.trial106.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5312.gab 845 kB/s | 57 kB 00:00 2026-02-26T03:15:58.566 INFO:teuthology.orchestra.run.trial106.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5312.gabffb617. 884 kB/s | 184 kB 00:00 2026-02-26T03:16:01.828 INFO:teuthology.orchestra.run.trial106.stdout:(8/96): ceph-mon-20.3.0-5312.gabffb617.el9.x86_ 413 kB/s | 4.9 MB 00:12 2026-02-26T03:16:11.771 INFO:teuthology.orchestra.run.trial106.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 761 kB/s | 7.4 MB 00:09 2026-02-26T03:16:11.838 INFO:teuthology.orchestra.run.trial106.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5312.gabffb6 327 kB/s | 22 kB 00:00 2026-02-26T03:16:12.326 INFO:teuthology.orchestra.run.trial106.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5312.gabf 608 kB/s | 297 kB 00:00 2026-02-26T03:16:12.470 INFO:teuthology.orchestra.run.trial106.stdout:(12/96): ceph-mgr-rook-20.3.0-5312.gabffb617.el 350 kB/s | 50 kB 00:00 2026-02-26T03:16:12.541 INFO:teuthology.orchestra.run.trial106.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5312.gab 244 kB/s | 17 kB 00:00 2026-02-26T03:16:12.964 INFO:teuthology.orchestra.run.trial106.stdout:(14/96): ceph-volume-20.3.0-5312.gabffb617.el9. 712 kB/s | 301 kB 00:00 2026-02-26T03:16:14.185 INFO:teuthology.orchestra.run.trial106.stdout:(15/96): cephadm-20.3.0-5312.gabffb617.el9.noar 852 kB/s | 1.0 MB 00:01 2026-02-26T03:16:14.331 INFO:teuthology.orchestra.run.trial106.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.3 MB/s | 351 kB 00:00 2026-02-26T03:16:14.354 INFO:teuthology.orchestra.run.trial106.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.7 MB/s | 40 kB 00:00 2026-02-26T03:16:14.378 INFO:teuthology.orchestra.run.trial106.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.0 MB/s | 72 kB 00:00 2026-02-26T03:16:14.435 INFO:teuthology.orchestra.run.trial106.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 14 MB/s | 794 kB 00:00 2026-02-26T03:16:14.462 INFO:teuthology.orchestra.run.trial106.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 6.9 MB/s | 184 kB 00:00 2026-02-26T03:16:14.486 INFO:teuthology.orchestra.run.trial106.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-26T03:16:14.532 INFO:teuthology.orchestra.run.trial106.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 27 MB/s | 1.2 MB 00:00 2026-02-26T03:16:14.557 INFO:teuthology.orchestra.run.trial106.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.2 MB/s | 106 kB 00:00 2026-02-26T03:16:14.579 INFO:teuthology.orchestra.run.trial106.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.9 MB/s | 135 kB 00:00 2026-02-26T03:16:14.603 INFO:teuthology.orchestra.run.trial106.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-26T03:16:14.630 INFO:teuthology.orchestra.run.trial106.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 4.6 MB/s | 126 kB 00:00 2026-02-26T03:16:14.657 INFO:teuthology.orchestra.run.trial106.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.0 MB/s | 218 kB 00:00 2026-02-26T03:16:14.681 INFO:teuthology.orchestra.run.trial106.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.4 MB/s | 182 kB 00:00 2026-02-26T03:16:14.705 INFO:teuthology.orchestra.run.trial106.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-26T03:16:19.791 INFO:teuthology.orchestra.run.trial106.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 5.8 kB/s | 30 kB 00:05 2026-02-26T03:16:20.013 INFO:teuthology.orchestra.run.trial106.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 13 MB/s | 3.0 MB 00:00 2026-02-26T03:16:20.078 INFO:teuthology.orchestra.run.trial106.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 231 kB/s | 15 kB 00:00 2026-02-26T03:16:20.204 INFO:teuthology.orchestra.run.trial106.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 1.9 MB/s | 246 kB 00:00 2026-02-26T03:16:20.312 INFO:teuthology.orchestra.run.trial106.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2026-02-26T03:16:20.399 INFO:teuthology.orchestra.run.trial106.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 483 kB/s | 42 kB 00:00 2026-02-26T03:16:20.686 INFO:teuthology.orchestra.run.trial106.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 18 MB/s | 5.3 MB 00:00 2026-02-26T03:16:20.995 INFO:teuthology.orchestra.run.trial106.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2026-02-26T03:16:21.126 INFO:teuthology.orchestra.run.trial106.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2026-02-26T03:16:21.263 INFO:teuthology.orchestra.run.trial106.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 1.8 MB/s | 249 kB 00:00 2026-02-26T03:16:21.383 INFO:teuthology.orchestra.run.trial106.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 397 kB/s | 48 kB 00:00 2026-02-26T03:16:21.493 INFO:teuthology.orchestra.run.trial106.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 1.6 MB/s | 177 kB 00:00 2026-02-26T03:16:21.667 INFO:teuthology.orchestra.run.trial106.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 7.1 MB/s | 1.2 MB 00:00 2026-02-26T03:16:21.732 INFO:teuthology.orchestra.run.trial106.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 534 kB/s | 35 kB 00:00 2026-02-26T03:16:22.040 INFO:teuthology.orchestra.run.trial106.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 20 MB/s | 6.1 MB 00:00 2026-02-26T03:16:22.189 INFO:teuthology.orchestra.run.trial106.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 2.9 MB/s | 442 kB 00:00 2026-02-26T03:16:22.275 INFO:teuthology.orchestra.run.trial106.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 893 kB/s | 77 kB 00:00 2026-02-26T03:16:22.405 INFO:teuthology.orchestra.run.trial106.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 2.0 MB/s | 267 kB 00:00 2026-02-26T03:16:22.514 INFO:teuthology.orchestra.run.trial106.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.4 MB/s | 157 kB 00:00 2026-02-26T03:16:22.642 INFO:teuthology.orchestra.run.trial106.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 2.1 MB/s | 277 kB 00:00 2026-02-26T03:16:22.730 INFO:teuthology.orchestra.run.trial106.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 607 kB/s | 54 kB 00:00 2026-02-26T03:16:23.289 INFO:teuthology.orchestra.run.trial106.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 35 MB/s | 19 MB 00:00 2026-02-26T03:16:23.375 INFO:teuthology.orchestra.run.trial106.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 482 kB/s | 42 kB 00:00 2026-02-26T03:16:23.488 INFO:teuthology.orchestra.run.trial106.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 1.6 MB/s | 189 kB 00:00 2026-02-26T03:16:23.574 INFO:teuthology.orchestra.run.trial106.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.0 MB/s | 90 kB 00:00 2026-02-26T03:16:24.131 INFO:teuthology.orchestra.run.trial106.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 40 kB/s | 22 kB 00:00 2026-02-26T03:16:24.605 INFO:teuthology.orchestra.run.trial106.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 1.8 MB/s | 862 kB 00:00 2026-02-26T03:16:25.066 INFO:teuthology.orchestra.run.trial106.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.2 MB/s | 551 kB 00:00 2026-02-26T03:16:25.128 INFO:teuthology.orchestra.run.trial106.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 315 kB/s | 19 kB 00:00 2026-02-26T03:16:25.200 INFO:teuthology.orchestra.run.trial106.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2026-02-26T03:16:25.303 INFO:teuthology.orchestra.run.trial106.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 5.2 MB/s | 548 kB 00:00 2026-02-26T03:16:25.366 INFO:teuthology.orchestra.run.trial106.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 468 kB/s | 29 kB 00:00 2026-02-26T03:16:25.431 INFO:teuthology.orchestra.run.trial106.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 928 kB/s | 60 kB 00:00 2026-02-26T03:16:25.495 INFO:teuthology.orchestra.run.trial106.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 682 kB/s | 43 kB 00:00 2026-02-26T03:16:25.558 INFO:teuthology.orchestra.run.trial106.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 513 kB/s | 32 kB 00:00 2026-02-26T03:16:25.620 INFO:teuthology.orchestra.run.trial106.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 227 kB/s | 14 kB 00:00 2026-02-26T03:16:25.693 INFO:teuthology.orchestra.run.trial106.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 2.3 MB/s | 173 kB 00:00 2026-02-26T03:16:25.777 INFO:teuthology.orchestra.run.trial106.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2026-02-26T03:16:25.855 INFO:teuthology.orchestra.run.trial106.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 3.2 MB/s | 254 kB 00:00 2026-02-26T03:16:26.042 INFO:teuthology.orchestra.run.trial106.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2026-02-26T03:16:26.107 INFO:teuthology.orchestra.run.trial106.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 2.2 MB/s | 144 kB 00:00 2026-02-26T03:16:26.171 INFO:teuthology.orchestra.run.trial106.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 2.1 MB/s | 139 kB 00:00 2026-02-26T03:16:26.233 INFO:teuthology.orchestra.run.trial106.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 896 kB/s | 56 kB 00:00 2026-02-26T03:16:26.295 INFO:teuthology.orchestra.run.trial106.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 173 kB/s | 11 kB 00:00 2026-02-26T03:16:26.357 INFO:teuthology.orchestra.run.trial106.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 286 kB/s | 18 kB 00:00 2026-02-26T03:16:26.419 INFO:teuthology.orchestra.run.trial106.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 374 kB/s | 23 kB 00:00 2026-02-26T03:16:26.481 INFO:teuthology.orchestra.run.trial106.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 317 kB/s | 20 kB 00:00 2026-02-26T03:16:26.543 INFO:teuthology.orchestra.run.trial106.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 314 kB/s | 19 kB 00:00 2026-02-26T03:16:26.605 INFO:teuthology.orchestra.run.trial106.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 426 kB/s | 26 kB 00:00 2026-02-26T03:16:26.687 INFO:teuthology.orchestra.run.trial106.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 13 MB/s | 1.0 MB 00:00 2026-02-26T03:16:26.750 INFO:teuthology.orchestra.run.trial106.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2026-02-26T03:16:26.813 INFO:teuthology.orchestra.run.trial106.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2026-02-26T03:16:26.876 INFO:teuthology.orchestra.run.trial106.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 922 kB/s | 58 kB 00:00 2026-02-26T03:16:26.937 INFO:teuthology.orchestra.run.trial106.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 266 kB/s | 16 kB 00:00 2026-02-26T03:16:27.000 INFO:teuthology.orchestra.run.trial106.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.4 MB/s | 90 kB 00:00 2026-02-26T03:16:27.062 INFO:teuthology.orchestra.run.trial106.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 496 kB/s | 31 kB 00:00 2026-02-26T03:16:27.126 INFO:teuthology.orchestra.run.trial106.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 2.9 MB/s | 188 kB 00:00 2026-02-26T03:16:27.189 INFO:teuthology.orchestra.run.trial106.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 945 kB/s | 59 kB 00:00 2026-02-26T03:16:27.253 INFO:teuthology.orchestra.run.trial106.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 1.9 MB/s | 125 kB 00:00 2026-02-26T03:16:27.315 INFO:teuthology.orchestra.run.trial106.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 575 kB/s | 36 kB 00:00 2026-02-26T03:16:27.378 INFO:teuthology.orchestra.run.trial106.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 1.3 MB/s | 86 kB 00:00 2026-02-26T03:16:27.440 INFO:teuthology.orchestra.run.trial106.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 1.4 MB/s | 90 kB 00:00 2026-02-26T03:16:27.502 INFO:teuthology.orchestra.run.trial106.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 776 kB/s | 48 kB 00:00 2026-02-26T03:16:27.564 INFO:teuthology.orchestra.run.trial106.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 358 kB/s | 22 kB 00:00 2026-02-26T03:16:27.626 INFO:teuthology.orchestra.run.trial106.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 323 kB/s | 20 kB 00:00 2026-02-26T03:17:07.019 INFO:teuthology.orchestra.run.trial106.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5312.gabffb6 757 kB/s | 51 MB 01:08 2026-02-26T03:18:04.999 INFO:teuthology.orchestra.run.trial106.stdout:(96/96): ceph-osd-crimson-20.3.0-5312.gabffb617 663 kB/s | 87 MB 02:13 2026-02-26T03:18:05.001 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:05.001 INFO:teuthology.orchestra.run.trial106.stdout:Total 1.7 MB/s | 225 MB 02:15 2026-02-26T03:18:05.379 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:05.418 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:05.418 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:06.220 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:06.221 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:07.050 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:07.052 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-26T03:18:07.056 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-26T03:18:07.077 INFO:teuthology.orchestra.run.trial106.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-26T03:18:07.110 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-26T03:18:07.116 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-26T03:18:07.119 INFO:teuthology.orchestra.run.trial106.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-26T03:18:07.124 INFO:teuthology.orchestra.run.trial106.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-26T03:18:07.147 INFO:teuthology.orchestra.run.trial106.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-26T03:18:07.159 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-26T03:18:07.162 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-26T03:18:07.168 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-26T03:18:07.170 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-26T03:18:07.182 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-26T03:18:07.187 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-26T03:18:07.211 INFO:teuthology.orchestra.run.trial106.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-26T03:18:07.214 INFO:teuthology.orchestra.run.trial106.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-26T03:18:07.264 INFO:teuthology.orchestra.run.trial106.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-26T03:18:07.277 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-26T03:18:07.279 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-26T03:18:07.284 INFO:teuthology.orchestra.run.trial106.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-26T03:18:07.289 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-26T03:18:07.294 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-26T03:18:07.297 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-26T03:18:07.311 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-26T03:18:07.316 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-26T03:18:07.321 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-26T03:18:07.332 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-26T03:18:07.341 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-26T03:18:07.347 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-26T03:18:07.366 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-26T03:18:07.377 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-26T03:18:07.393 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-26T03:18:07.397 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-26T03:18:07.405 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-26T03:18:07.445 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-26T03:18:07.767 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-26T03:18:07.777 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-26T03:18:07.781 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-26T03:18:07.787 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-26T03:18:07.789 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-26T03:18:07.815 INFO:teuthology.orchestra.run.trial106.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-26T03:18:07.859 INFO:teuthology.orchestra.run.trial106.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-26T03:18:07.868 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-26T03:18:07.873 INFO:teuthology.orchestra.run.trial106.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-26T03:18:07.877 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-26T03:18:07.883 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-26T03:18:07.886 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-26T03:18:07.893 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-26T03:18:07.896 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-26T03:18:07.924 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-26T03:18:07.934 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-26T03:18:07.939 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-26T03:18:07.947 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-26T03:18:07.981 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-26T03:18:08.203 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-26T03:18:08.227 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-26T03:18:08.230 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-26T03:18:08.289 INFO:teuthology.orchestra.run.trial106.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-26T03:18:08.291 INFO:teuthology.orchestra.run.trial106.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-26T03:18:08.312 INFO:teuthology.orchestra.run.trial106.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-26T03:18:08.637 INFO:teuthology.orchestra.run.trial106.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-26T03:18:08.709 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-26T03:18:09.379 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-26T03:18:09.395 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-26T03:18:09.405 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-26T03:18:09.412 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-26T03:18:09.425 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-26T03:18:09.440 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-26T03:18:09.516 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-26T03:18:09.525 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-26T03:18:09.548 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-26T03:18:09.577 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-26T03:18:09.628 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-26T03:18:09.633 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-26T03:18:09.637 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-26T03:18:09.638 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-26T03:18:09.656 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:18:09.656 INFO:teuthology.orchestra.run.trial106.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:18:09.657 INFO:teuthology.orchestra.run.trial106.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:18:09.657 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:09.664 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:18:09.688 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:18:09.688 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:18:09.688 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:09.722 INFO:teuthology.orchestra.run.trial106.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-26T03:18:10.309 INFO:teuthology.orchestra.run.trial106.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-26T03:18:15.640 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 80/96 2026-02-26T03:18:15.704 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 81/96 2026-02-26T03:18:15.706 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:18:15.719 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:18:15.719 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:18:15.720 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:18:15.720 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:15.730 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:18:15.754 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:18:15.755 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:18:15.755 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:18:15.755 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:18:15.755 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:18:15.756 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:15.813 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:18:15.815 INFO:teuthology.orchestra.run.trial106.stdout: Installing : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:18:15.824 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 85/96 2026-02-26T03:18:15.848 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 86/96 2026-02-26T03:18:15.849 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:18:18.729 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:18:18.735 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:18:19.190 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:18:19.190 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:18:19.203 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:18:19.203 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:18:19.265 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:18:19.305 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 91/96 2026-02-26T03:18:19.306 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:18:19.326 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:18:19.326 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:18:19.326 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:18:19.326 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:18:19.327 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:18:19.327 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:19.338 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:18:19.555 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:18:19.557 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:18:19.578 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:18:19.578 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:18:19.578 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:18:19.578 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:18:19.578 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:18:19.578 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:19.660 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:18:19.681 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:18:19.682 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:18:19.682 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:18:19.682 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:18:19.682 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:18:19.683 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:19.683 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:18:19.688 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:18:19.713 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 1/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 3/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 4/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 5/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 6/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 7/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 8/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 9/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 10/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 11/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 12/96 2026-02-26T03:18:21.085 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 13/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 14/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 15/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 16/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 17/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-26T03:18:21.086 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-26T03:18:21.090 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-26T03:18:21.090 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-26T03:18:21.090 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-26T03:18:21.090 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-26T03:18:21.091 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-26T03:18:21.092 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-26T03:18:21.093 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-26T03:18:21.094 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-26T03:18:21.094 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-26T03:18:21.094 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-26T03:18:21.094 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-26T03:18:21.158 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-26T03:18:21.158 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.159 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:18:21.160 INFO:teuthology.orchestra.run.trial106.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:18:21.161 INFO:teuthology.orchestra.run.trial106.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:18:21.162 INFO:teuthology.orchestra.run.trial106.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:18:21.163 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:18:21.164 INFO:teuthology.orchestra.run.trial106.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:18:21.165 INFO:teuthology.orchestra.run.trial106.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:18:21.166 INFO:teuthology.orchestra.run.trial106.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:18:21.167 INFO:teuthology.orchestra.run.trial106.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:18:21.167 INFO:teuthology.orchestra.run.trial106.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:18:21.167 INFO:teuthology.orchestra.run.trial106.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:18:21.167 INFO:teuthology.orchestra.run.trial106.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:18:21.167 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:21.167 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:21.225 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-base 2026-02-26T03:18:21.540 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:15 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:21.605 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:21.624 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:21.624 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:21.625 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:21.646 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install cephadm 2026-02-26T03:18:21.959 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:15 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:22.025 INFO:teuthology.orchestra.run.trial106.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:18:22.043 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:22.044 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:22.044 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:22.065 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:18:22.375 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:16 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:22.459 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repo Size 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.gabffb617.el9 ceph 149 k 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:Install 1 Package 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 149 k 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 455 k 2026-02-26T03:18:22.460 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:18:23.223 INFO:teuthology.orchestra.run.trial106.stdout:ceph-immutable-object-cache-20.3.0-5312.gabffb6 196 kB/s | 149 kB 00:00 2026-02-26T03:18:23.224 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:23.224 INFO:teuthology.orchestra.run.trial106.stdout:Total 196 kB/s | 149 kB 00:00 2026-02-26T03:18:23.224 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:23.228 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:23.228 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:23.259 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:23.259 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:23.315 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:23.316 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:18:23.339 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:18:23.340 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:18:23.340 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:18:23.340 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:23.641 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:18:23.641 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:23.642 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:23.642 INFO:teuthology.orchestra.run.trial106.stdout: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:23.642 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:23.642 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:23.668 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr 2026-02-26T03:18:23.989 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:17 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:24.055 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:24.074 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:24.074 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:24.075 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:24.096 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:18:24.415 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:18 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:24.482 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:18:24.501 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:24.501 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:24.501 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:24.523 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:18:24.834 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:18 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:24.898 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:18:24.917 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:24.918 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:24.918 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:24.938 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:18:25.249 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:19 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:25.314 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:18:25.333 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:25.333 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:25.333 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:25.354 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:18:25.672 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:19 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:25.737 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:18:25.755 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:25.756 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:25.756 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:25.777 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-osd 2026-02-26T03:18:26.088 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:20 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:26.154 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:26.173 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:26.174 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:26.174 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:26.195 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-osd-classic 2026-02-26T03:18:26.513 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:20 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:26.579 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:26.598 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:26.599 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:26.599 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:26.619 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-fuse 2026-02-26T03:18:26.936 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:20 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:27.022 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:27.022 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:27.022 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repo Size 2026-02-26T03:18:27.022 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:27.022 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:27.022 INFO:teuthology.orchestra.run.trial106.stdout: ceph-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 914 k 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout:Install 1 Package 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 914 k 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 2.9 M 2026-02-26T03:18:27.023 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:18:28.357 INFO:teuthology.orchestra.run.trial106.stdout:ceph-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 685 kB/s | 914 kB 00:01 2026-02-26T03:18:28.358 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:28.358 INFO:teuthology.orchestra.run.trial106.stdout:Total 684 kB/s | 914 kB 00:01 2026-02-26T03:18:28.358 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:28.363 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:28.363 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:28.396 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:28.396 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:28.488 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:28.505 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:28.748 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:28.811 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:28.811 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:28.811 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:28.811 INFO:teuthology.orchestra.run.trial106.stdout: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:28.811 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:28.811 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:28.840 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-volume 2026-02-26T03:18:29.158 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:23 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:29.223 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:18:29.242 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:29.242 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:29.242 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:29.263 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install librados-devel 2026-02-26T03:18:29.580 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:23 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:29.664 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:29.664 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:29.664 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repo Size 2026-02-26T03:18:29.664 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:29.664 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:29.664 INFO:teuthology.orchestra.run.trial106.stdout: librados-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 124 k 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout:Install 1 Package 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 124 k 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 434 k 2026-02-26T03:18:29.665 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:18:30.248 INFO:teuthology.orchestra.run.trial106.stdout:librados-devel-20.3.0-5312.gabffb617.el9.x86_64 212 kB/s | 124 kB 00:00 2026-02-26T03:18:30.249 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:30.250 INFO:teuthology.orchestra.run.trial106.stdout:Total 211 kB/s | 124 kB 00:00 2026-02-26T03:18:30.250 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:30.254 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:30.254 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:30.286 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:30.286 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:30.340 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:30.357 INFO:teuthology.orchestra.run.trial106.stdout: Installing : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:30.445 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:30.507 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:30.507 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:30.507 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:30.507 INFO:teuthology.orchestra.run.trial106.stdout: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:30.508 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:30.508 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:30.535 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install libcephfs2 2026-02-26T03:18:30.849 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:24 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:30.914 INFO:teuthology.orchestra.run.trial106.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:30.933 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:30.934 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:30.934 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:30.955 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install libcephfs-devel 2026-02-26T03:18:31.269 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:25 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:31.353 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repo Size 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:31.354 INFO:teuthology.orchestra.run.trial106.stdout:Install 1 Package 2026-02-26T03:18:31.355 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:31.355 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 37 k 2026-02-26T03:18:31.355 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 171 k 2026-02-26T03:18:31.355 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:18:31.700 INFO:teuthology.orchestra.run.trial106.stdout:libcephfs-devel-20.3.0-5312.gabffb617.el9.x86_6 108 kB/s | 37 kB 00:00 2026-02-26T03:18:31.700 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:31.700 INFO:teuthology.orchestra.run.trial106.stdout:Total 108 kB/s | 37 kB 00:00 2026-02-26T03:18:31.701 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:31.702 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:31.702 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:31.704 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:31.704 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:31.727 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:31.744 INFO:teuthology.orchestra.run.trial106.stdout: Installing : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:31.805 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:31.866 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:31.866 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:31.866 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:31.867 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:31.867 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:31.867 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:31.895 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install librados2 2026-02-26T03:18:32.217 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:26 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:32.284 INFO:teuthology.orchestra.run.trial106.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:32.301 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:32.302 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:32.302 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:32.324 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install librbd1 2026-02-26T03:18:32.640 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:26 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:32.705 INFO:teuthology.orchestra.run.trial106.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:32.723 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:32.723 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:32.723 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:32.745 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-rados 2026-02-26T03:18:33.058 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:27 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:33.123 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:33.141 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:33.141 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:33.141 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:33.163 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-rgw 2026-02-26T03:18:33.481 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:27 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:33.546 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:33.563 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:33.564 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:33.564 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:33.585 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-cephfs 2026-02-26T03:18:33.900 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:27 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:33.965 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:33.982 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:33.983 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:33.983 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:34.005 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-rbd 2026-02-26T03:18:34.321 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:28 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:34.386 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:34.404 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:34.404 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:34.405 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:34.426 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install rbd-fuse 2026-02-26T03:18:34.738 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:28 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:34.822 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout: Package Architecture Version Repository Size 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout: rbd-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 k 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:Install 1 Package 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 87 k 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 236 k 2026-02-26T03:18:34.823 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:18:35.248 INFO:teuthology.orchestra.run.trial106.stdout:rbd-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 204 kB/s | 87 kB 00:00 2026-02-26T03:18:35.248 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:35.248 INFO:teuthology.orchestra.run.trial106.stdout:Total 204 kB/s | 87 kB 00:00 2026-02-26T03:18:35.248 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:35.252 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:35.252 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:35.284 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:35.284 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:35.335 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:35.353 INFO:teuthology.orchestra.run.trial106.stdout: Installing : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:35.437 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:35.499 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:35.499 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:35.499 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:35.499 INFO:teuthology.orchestra.run.trial106.stdout: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:35.499 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:35.499 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:35.527 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install rbd-mirror 2026-02-26T03:18:35.840 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:29 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repo Size 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout: rbd-mirror x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.6 M 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:Install 1 Package 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 2.6 M 2026-02-26T03:18:35.923 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 11 M 2026-02-26T03:18:35.924 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:18:37.440 INFO:teuthology.orchestra.run.trial106.stdout:rbd-mirror-20.3.0-5312.gabffb617.el9.x86_64.rpm 1.7 MB/s | 2.6 MB 00:01 2026-02-26T03:18:37.441 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:37.441 INFO:teuthology.orchestra.run.trial106.stdout:Total 1.7 MB/s | 2.6 MB 00:01 2026-02-26T03:18:37.441 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:37.446 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:37.447 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:37.485 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:37.485 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:37.648 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:37.650 INFO:teuthology.orchestra.run.trial106.stdout: Installing : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:37.672 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:37.672 INFO:teuthology.orchestra.run.trial106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:18:37.672 INFO:teuthology.orchestra.run.trial106.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:18:37.672 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:18:37.673 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:18:37.673 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:37.993 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:37.993 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:37.993 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:37.993 INFO:teuthology.orchestra.run.trial106.stdout: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:37.993 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:37.993 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:38.021 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install rbd-nbd 2026-02-26T03:18:38.337 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:32 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:38.421 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:38.421 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:38.421 INFO:teuthology.orchestra.run.trial106.stdout: Package Architecture Version Repository Size 2026-02-26T03:18:38.421 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout: rbd-nbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 166 k 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout:Install 1 Package 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 166 k 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 496 k 2026-02-26T03:18:38.422 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:18:39.215 INFO:teuthology.orchestra.run.trial106.stdout:rbd-nbd-20.3.0-5312.gabffb617.el9.x86_64.rpm 209 kB/s | 166 kB 00:00 2026-02-26T03:18:39.215 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:18:39.216 INFO:teuthology.orchestra.run.trial106.stdout:Total 209 kB/s | 166 kB 00:00 2026-02-26T03:18:39.216 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:18:39.221 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:18:39.221 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:18:39.252 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:18:39.253 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:18:39.309 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:18:39.327 INFO:teuthology.orchestra.run.trial106.stdout: Installing : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:39.416 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:39.478 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:18:39.478 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:39.478 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:18:39.478 INFO:teuthology.orchestra.run.trial106.stdout: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:18:39.478 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:39.478 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:39.506 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-osd-crimson 2026-02-26T03:18:39.827 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:33 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:39.892 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:18:39.910 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:39.910 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:18:39.910 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:18:39.932 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-26T03:18:40.250 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:05:34 ago on Thu 26 Feb 2026 03:13:06 AM UTC. 2026-02-26T03:18:40.333 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout: Package Arch Version Repo Size 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout:Installing: 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.gabffb617.el9 ceph 429 M 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout:Installing weak dependencies: 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout: ceph-debugsource x86_64 2:20.3.0-5312.gabffb617.el9 ceph 12 M 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout:Transaction Summary 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout:================================================================================ 2026-02-26T03:18:40.334 INFO:teuthology.orchestra.run.trial106.stdout:Install 2 Packages 2026-02-26T03:18:40.335 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:18:40.335 INFO:teuthology.orchestra.run.trial106.stdout:Total download size: 441 M 2026-02-26T03:18:40.335 INFO:teuthology.orchestra.run.trial106.stdout:Installed size: 2.7 G 2026-02-26T03:18:40.335 INFO:teuthology.orchestra.run.trial106.stdout:Downloading Packages: 2026-02-26T03:19:02.345 INFO:teuthology.orchestra.run.trial106.stdout:(1/2): ceph-debugsource-20.3.0-5312.gabffb617.e 560 kB/s | 12 MB 00:22 2026-02-26T03:29:28.879 INFO:teuthology.orchestra.run.trial106.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5312.g 677 kB/s | 429 MB 10:48 2026-02-26T03:29:28.879 INFO:teuthology.orchestra.run.trial106.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:29:28.880 INFO:teuthology.orchestra.run.trial106.stdout:Total 696 kB/s | 441 MB 10:48 2026-02-26T03:29:28.880 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction check 2026-02-26T03:29:28.885 INFO:teuthology.orchestra.run.trial106.stdout:Transaction check succeeded. 2026-02-26T03:29:28.885 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction test 2026-02-26T03:29:30.247 INFO:teuthology.orchestra.run.trial106.stdout:Transaction test succeeded. 2026-02-26T03:29:30.248 INFO:teuthology.orchestra.run.trial106.stdout:Running transaction 2026-02-26T03:29:33.832 INFO:teuthology.orchestra.run.trial106.stdout: Preparing : 1/1 2026-02-26T03:29:56.251 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:29:56.341 INFO:teuthology.orchestra.run.trial106.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:29:57.011 INFO:teuthology.orchestra.run.trial106.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:29:57.012 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:29:57.075 INFO:teuthology.orchestra.run.trial106.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:29:57.075 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:29:57.075 INFO:teuthology.orchestra.run.trial106.stdout:Installed: 2026-02-26T03:29:57.075 INFO:teuthology.orchestra.run.trial106.stdout: ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:29:57.075 INFO:teuthology.orchestra.run.trial106.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:29:57.075 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:29:57.075 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:29:57.150 DEBUG:teuthology.parallel:result is None 2026-02-26T03:29:57.151 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:29:57.331 DEBUG:teuthology.orchestra.run.trial106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:29:57.353 INFO:teuthology.orchestra.run.trial106.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:29:57.354 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:29:57.354 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:29:57.356 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:29:57.356 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:29:57.357 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:29:57.419 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:29:57.420 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:29:57.420 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:29:57.483 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:29:57.547 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:29:57.547 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:29:57.547 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:29:57.611 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:29:57.674 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:29:57.675 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:29:57.675 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:29:57.739 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:29:57.802 INFO:teuthology.run_tasks:Running task pexec... 2026-02-26T03:29:57.815 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-26T03:29:57.815 DEBUG:teuthology.orchestra.run.trial106:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-26T03:29:57.844 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:29:57.844 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-26T03:29:57.895 DEBUG:teuthology.parallel:result is None 2026-02-26T03:29:57.895 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:29:57.909 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:29:57.909 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:29:57.909 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:29:57.910 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:29:57.918 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:29:57.919 INFO:teuthology.task.install:extra packages: [] 2026-02-26T03:29:57.919 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:29:57.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:29:58.080 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:29:58.080 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:29:58.755 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:29:58.756 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:29:58.756 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:29:58.782 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-26T03:29:58.782 DEBUG:teuthology.orchestra.run.trial106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:29:58.847 DEBUG:teuthology.orchestra.run.trial106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:29:58.912 DEBUG:teuthology.orchestra.run.trial106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:29:58.968 INFO:teuthology.orchestra.run.trial106.stdout:check_obsoletes = 1 2026-02-26T03:29:58.977 DEBUG:teuthology.orchestra.run.trial106:> sudo yum clean all 2026-02-26T03:29:59.140 INFO:teuthology.orchestra.run.trial106.stdout:56 files removed 2026-02-26T03:29:59.157 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-radosgw 2026-02-26T03:30:00.635 INFO:teuthology.orchestra.run.trial106.stdout:ceph packages for x86_64 71 kB/s | 94 kB 00:01 2026-02-26T03:30:01.314 INFO:teuthology.orchestra.run.trial106.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-26T03:30:01.914 INFO:teuthology.orchestra.run.trial106.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:30:28.595 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - BaseOS 342 kB/s | 8.9 MB 00:26 2026-02-26T03:31:01.234 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - AppStream 860 kB/s | 27 MB 00:32 2026-02-26T03:31:05.602 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 8.0 MB 00:01 2026-02-26T03:31:06.431 INFO:teuthology.orchestra.run.trial106.stdout:CentOS Stream 9 - Extras packages 225 kB/s | 20 kB 00:00 2026-02-26T03:31:06.783 INFO:teuthology.orchestra.run.trial106.stdout:Extra Packages for Enterprise Linux 72 MB/s | 20 MB 00:00 2026-02-26T03:31:09.959 INFO:teuthology.orchestra.run.trial106.stdout:lab-extras 2.2 MB/s | 50 kB 00:00 2026-02-26T03:31:11.171 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:11.189 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:11.190 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:11.190 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:11.213 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-test 2026-02-26T03:31:11.532 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:11.596 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:11.614 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:11.615 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:11.615 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:11.637 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph 2026-02-26T03:31:11.957 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:12.022 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:12.039 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:12.040 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:12.040 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:12.062 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-base 2026-02-26T03:31:12.375 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:12.439 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:12.458 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:12.458 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:12.458 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:12.479 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install cephadm 2026-02-26T03:31:12.795 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:12.860 INFO:teuthology.orchestra.run.trial106.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:12.878 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:12.878 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:12.878 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:12.900 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:31:13.213 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:13.278 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:13.296 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:13.297 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:13.297 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:13.318 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr 2026-02-26T03:31:13.637 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:13.701 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:13.719 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:13.720 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:13.720 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:13.742 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:31:14.059 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:14.124 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:14.142 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:14.142 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:14.142 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:14.164 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:31:14.485 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:14.550 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:14.569 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:14.570 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:14.570 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:14.590 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:31:14.908 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:14.974 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:14.991 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:14.992 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:14.992 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:15.014 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:31:15.335 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:15.401 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:15.418 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:15.419 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:15.419 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:15.440 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-osd 2026-02-26T03:31:15.757 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:15.821 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:15.839 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:15.839 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:15.839 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:15.860 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-osd-classic 2026-02-26T03:31:16.178 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:16.243 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:16.261 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:16.261 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:16.262 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:16.284 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-fuse 2026-02-26T03:31:16.604 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:16.668 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:16.686 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:16.687 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:16.687 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:16.708 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install ceph-volume 2026-02-26T03:31:17.028 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:17.093 INFO:teuthology.orchestra.run.trial106.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:17.111 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:17.112 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:17.112 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:17.134 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install librados-devel 2026-02-26T03:31:17.454 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:17.519 INFO:teuthology.orchestra.run.trial106.stdout:Package librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:17.537 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:17.537 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:17.537 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:17.559 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install libcephfs2 2026-02-26T03:31:17.880 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:17.945 INFO:teuthology.orchestra.run.trial106.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:17.963 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:17.964 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:17.964 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:17.986 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install libcephfs-devel 2026-02-26T03:31:18.300 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:18.365 INFO:teuthology.orchestra.run.trial106.stdout:Package libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:18.383 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:18.384 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:18.384 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:18.406 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install librados2 2026-02-26T03:31:18.722 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:18.787 INFO:teuthology.orchestra.run.trial106.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:18.804 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:18.805 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:18.805 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:18.826 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install librbd1 2026-02-26T03:31:19.144 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:19.209 INFO:teuthology.orchestra.run.trial106.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:19.226 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:19.227 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:19.227 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:19.249 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-rados 2026-02-26T03:31:19.570 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:19.635 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:19.653 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:19.654 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:19.654 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:19.675 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-rgw 2026-02-26T03:31:19.996 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:20.062 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:20.080 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:20.080 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:20.080 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:20.103 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-cephfs 2026-02-26T03:31:20.423 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:20.488 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:20.506 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:20.507 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:20.507 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:20.529 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install python3-rbd 2026-02-26T03:31:20.853 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:20.919 INFO:teuthology.orchestra.run.trial106.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:20.937 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:20.937 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:20.938 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:20.960 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install rbd-fuse 2026-02-26T03:31:21.277 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:21.342 INFO:teuthology.orchestra.run.trial106.stdout:Package rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:21.359 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:21.360 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:21.360 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:21.382 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install rbd-mirror 2026-02-26T03:31:21.699 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:21.765 INFO:teuthology.orchestra.run.trial106.stdout:Package rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:21.783 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:21.784 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:21.784 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:21.805 DEBUG:teuthology.orchestra.run.trial106:> sudo yum -y install rbd-nbd 2026-02-26T03:31:22.123 INFO:teuthology.orchestra.run.trial106.stdout:Last metadata expiration check: 0:00:13 ago on Thu 26 Feb 2026 03:31:09 AM UTC. 2026-02-26T03:31:22.188 INFO:teuthology.orchestra.run.trial106.stdout:Package rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:22.206 INFO:teuthology.orchestra.run.trial106.stdout:Dependencies resolved. 2026-02-26T03:31:22.206 INFO:teuthology.orchestra.run.trial106.stdout:Nothing to do. 2026-02-26T03:31:22.207 INFO:teuthology.orchestra.run.trial106.stdout:Complete! 2026-02-26T03:31:22.232 DEBUG:teuthology.parallel:result is None 2026-02-26T03:31:22.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:31:22.405 DEBUG:teuthology.orchestra.run.trial106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:31:22.427 INFO:teuthology.orchestra.run.trial106.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:31:22.427 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:31:22.427 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:31:22.431 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-26T03:31:22.431 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:31:22.534 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:31:22.534 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod 777 /var/log/ceph 2026-02-26T03:31:22.559 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:31:22.559 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:31:22.622 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:31:22.622 DEBUG:teuthology.orchestra.run.trial106:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:31:22.698 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:31:22.698 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:31:22.699 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71180', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_omap_write}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '71180', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5117, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996', 'targets': {'trial106.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGu4IO27ushH8zqNwnk4x3940jSLIzQAUGkeVqcSExWs3YKuuTDXAKvrhc7NBdf95pbPfC/nb/9Pm9Vfe9zyJuA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'radosbench': {'clients': ['client.0'], 'objectsize': 4096, 'size': 4096, 'time': 300, 'write-omap': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_02:22:56', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:31:22.699 DEBUG:teuthology.orchestra.run.trial106:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:31:22.743 DEBUG:teuthology.orchestra.run.trial106:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:31:22.808 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:22.808 DEBUG:teuthology.orchestra.run.trial106:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:31:22.866 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:31:22.866 DEBUG:teuthology.orchestra.run.trial106:> stat /dev/vg_nvme/lv_1 2026-02-26T03:31:22.923 INFO:teuthology.orchestra.run.trial106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:31:22.923 INFO:teuthology.orchestra.run.trial106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:31:22.924 INFO:teuthology.orchestra.run.trial106.stdout:Device: 6h/6d Inode: 1298 Links: 1 2026-02-26T03:31:22.924 INFO:teuthology.orchestra.run.trial106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:31:22.924 INFO:teuthology.orchestra.run.trial106.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:31:22.924 INFO:teuthology.orchestra.run.trial106.stdout:Access: 2026-02-26 03:18:37.806293831 +0000 2026-02-26T03:31:22.924 INFO:teuthology.orchestra.run.trial106.stdout:Modify: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:22.924 INFO:teuthology.orchestra.run.trial106.stdout:Change: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:22.924 INFO:teuthology.orchestra.run.trial106.stdout: Birth: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:22.925 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:31:22.986 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records in 2026-02-26T03:31:22.987 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records out 2026-02-26T03:31:22.987 INFO:teuthology.orchestra.run.trial106.stderr:512 bytes copied, 0.000153224 s, 3.3 MB/s 2026-02-26T03:31:22.988 DEBUG:teuthology.orchestra.run.trial106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:31:23.045 DEBUG:teuthology.orchestra.run.trial106:> stat /dev/vg_nvme/lv_2 2026-02-26T03:31:23.101 INFO:teuthology.orchestra.run.trial106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout:Access: 2026-02-26 03:18:37.807293802 +0000 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout:Modify: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout:Change: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:23.102 INFO:teuthology.orchestra.run.trial106.stdout: Birth: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:23.103 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:31:23.165 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records in 2026-02-26T03:31:23.165 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records out 2026-02-26T03:31:23.165 INFO:teuthology.orchestra.run.trial106.stderr:512 bytes copied, 0.000151692 s, 3.4 MB/s 2026-02-26T03:31:23.167 DEBUG:teuthology.orchestra.run.trial106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:31:23.224 DEBUG:teuthology.orchestra.run.trial106:> stat /dev/vg_nvme/lv_3 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout:Device: 6h/6d Inode: 1290 Links: 1 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout:Access: 2026-02-26 03:18:37.807293802 +0000 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout:Modify: 2026-02-26 03:13:35.111014684 +0000 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout:Change: 2026-02-26 03:13:35.111014684 +0000 2026-02-26T03:31:23.281 INFO:teuthology.orchestra.run.trial106.stdout: Birth: 2026-02-26 03:13:35.111014684 +0000 2026-02-26T03:31:23.282 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:31:23.344 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records in 2026-02-26T03:31:23.344 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records out 2026-02-26T03:31:23.344 INFO:teuthology.orchestra.run.trial106.stderr:512 bytes copied, 0.000156981 s, 3.3 MB/s 2026-02-26T03:31:23.345 DEBUG:teuthology.orchestra.run.trial106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:31:23.402 DEBUG:teuthology.orchestra.run.trial106:> stat /dev/vg_nvme/lv_4 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout:Access: 2026-02-26 03:18:37.807293802 +0000 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout:Modify: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:23.459 INFO:teuthology.orchestra.run.trial106.stdout:Change: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:23.460 INFO:teuthology.orchestra.run.trial106.stdout: Birth: 2026-02-26 03:13:35.112014655 +0000 2026-02-26T03:31:23.460 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:31:23.522 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records in 2026-02-26T03:31:23.522 INFO:teuthology.orchestra.run.trial106.stderr:1+0 records out 2026-02-26T03:31:23.522 INFO:teuthology.orchestra.run.trial106.stderr:512 bytes copied, 0.000136503 s, 3.8 MB/s 2026-02-26T03:31:23.523 DEBUG:teuthology.orchestra.run.trial106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:31:23.579 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:31:23.580 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial106.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:31:23.580 INFO:tasks.ceph:Generating config... 2026-02-26T03:31:23.582 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-26T03:31:23.583 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-26T03:31:23.584 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-26T03:31:23.585 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-26T03:31:23.585 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-26T03:31:23.585 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-26T03:31:23.585 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] auth client required = none 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] auth service required = none 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] auth supported = none 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-26T03:31:23.585 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-26T03:31:23.586 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-26T03:31:23.587 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-26T03:31:23.588 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-26T03:31:23.589 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-26T03:31:23.590 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-26T03:31:23.590 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:31:23.590 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:31:23.590 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-26T03:31:23.590 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-26T03:31:23.590 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-26T03:31:23.591 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-26T03:31:23.591 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:31:23.591 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:31:23.660 INFO:teuthology.orchestra.run.trial106.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:31:23.663 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:31:23.747 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:31:23.811 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.106')] 2026-02-26T03:31:23.812 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.106', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:31:23.814 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:23.814 DEBUG:teuthology.orchestra.run.trial106:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:31:23.868 DEBUG:teuthology.orchestra.run.trial106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.106 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:31:23.949 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:23.947+0000 7f6f58e2bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:23.949 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:23.947+0000 7f6f58e2bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:23.949 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:23.948+0000 7f6f58e2bf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:monmaptool: generated fsid e84db318-d879-4a23-9cae-25a52f67fed2 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:setting min_mon_release = tentacle 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:epoch 0 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:fsid e84db318-d879-4a23-9cae-25a52f67fed2 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:last_changed 2026-02-26T03:31:23.949513+0000 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:created 2026-02-26T03:31:23.949513+0000 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:min_mon_release 20 (tentacle) 2026-02-26T03:31:23.951 INFO:teuthology.orchestra.run.trial106.stdout:election_strategy: 1 2026-02-26T03:31:23.952 INFO:teuthology.orchestra.run.trial106.stdout:0: [v2:10.20.193.106:3300/0,v1:10.20.193.106:6789/0] mon.a 2026-02-26T03:31:23.952 INFO:teuthology.orchestra.run.trial106.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:31:23.952 DEBUG:teuthology.orchestra.run.trial106:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:31:24.008 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e84db318-d879-4a23-9cae-25a52f67fed2... 2026-02-26T03:31:24.010 DEBUG:teuthology.orchestra.run.trial106:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:31:24.091 INFO:teuthology.orchestra.run.trial106.stdout:[global] 2026-02-26T03:31:24.091 INFO:teuthology.orchestra.run.trial106.stdout: chdir = "" 2026-02-26T03:31:24.091 INFO:teuthology.orchestra.run.trial106.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:31:24.091 INFO:teuthology.orchestra.run.trial106.stdout: auth supported = none 2026-02-26T03:31:24.091 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.091 INFO:teuthology.orchestra.run.trial106.stdout: filestore xattr use omap = true 2026-02-26T03:31:24.091 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: mon clock drift allowed = 1.000 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: osd crush chooseleaf type = 0 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: auth debug = true 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: ms die on old message = true 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: ms die on bug = true 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: mon pg warn max object skew = 0 2026-02-26T03:31:24.092 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: osd pool default size = 2 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon osd allow primary affinity = true 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon osd allow pg remap = true 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon warn on no sortbitwise = false 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon warn on too few osds = false 2026-02-26T03:31:24.093 INFO:teuthology.orchestra.run.trial106.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: mon_allow_pool_size_one = true 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: osd default data pool replay window = 5 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: mon allow pool delete = true 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: mon cluster log file level = debug 2026-02-26T03:31:24.094 INFO:teuthology.orchestra.run.trial106.stdout: debug asserts on shutdown = true 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: mon health detail to clog = false 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: mon host = 10.20.193.106 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: auth client required = none 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: auth cluster required = none 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: auth service required = none 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug asok = 0/0 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug auth = 0/0 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug bdev = 0/0 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug bluefs = 0/0 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug bluestore = 0/0 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug buffer = 0/0 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug civetweb = 0/0 2026-02-26T03:31:24.095 INFO:teuthology.orchestra.run.trial106.stdout: debug client = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug compressor = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug context = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug crush = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug crypto = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug dpdk = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug eventtrace = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug filer = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug filestore = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug finisher = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug fuse = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug heartbeatmap = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug javaclient = 0/0 2026-02-26T03:31:24.096 INFO:teuthology.orchestra.run.trial106.stdout: debug journal = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug journaler = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug kstore = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug leveldb = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug lockdep = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug mds = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug mds balancer = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug mds locker = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug mds log = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug mds log expire = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug mds migrator = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug memdb = 0/0 2026-02-26T03:31:24.097 INFO:teuthology.orchestra.run.trial106.stdout: debug mgr = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug mgrc = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug mon = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug monc = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug ms = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug objclass = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug objectcacher = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug objecter = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug optracker = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug osd = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug paxos = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug perfcounter = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug rados = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug rbd = 0/0 2026-02-26T03:31:24.098 INFO:teuthology.orchestra.run.trial106.stdout: debug rbd mirror = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug rbd replay = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug refs = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug rgw = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug rgw lifecycle = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug rgw notification = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug rgw sync = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug rocksdb = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug striper = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug throttle = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug timer = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: debug tp = 0/0 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:31:24.099 INFO:teuthology.orchestra.run.trial106.stdout: ms client mode = crc 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: ms cluster mode = crc 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: ms mon client mode = crc 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: ms mon cluster mode = crc 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: ms mon service mode = crc 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: ms service mode = crc 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: fsid = e84db318-d879-4a23-9cae-25a52f67fed2 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout:[osd] 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: osd journal size = 100 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: osd scrub load threshold = 5.0 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: osd scrub max interval = 600 2026-02-26T03:31:24.100 INFO:teuthology.orchestra.run.trial106.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd mclock skip benchmark = true 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd recover clone overlap = true 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd recovery max chunk = 1048576 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd debug shutdown = true 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd debug op order = true 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd debug verify stray on activate = true 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd debug trim objects = true 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd open classes on start = true 2026-02-26T03:31:24.101 INFO:teuthology.orchestra.run.trial106.stdout: osd debug pg log writeout = true 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: osd map max advance = 10 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: journal zero on create = true 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: filestore apply finisher threads = 3 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: bdev debug aio = true 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: osd debug misdirected ops = true 2026-02-26T03:31:24.102 INFO:teuthology.orchestra.run.trial106.stdout: debug monc = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: crimson cpu num = 4 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: crimson osd obc lru size = 10 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug filestore = 0/0 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug journal = 0/0 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug ms = 0/0 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug osd = 0/0 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore backref = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore cache = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore cleaner = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore device = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore epm = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore fixedkv tree = 20 2026-02-26T03:31:24.103 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore journal = 20 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore lba = 20 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore odata = 20 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore omap = 20 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore onode = 20 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore t = 20 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: debug seastore tm = 20 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: osd objectstore = seastore 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: osd shutdown pgref assert = True 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: seastore cachepin size pershard = 64M 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: seastore cachepin type = 2Q 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: seastore max concurrent transactions = 8 2026-02-26T03:31:24.104 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout:[mgr] 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: debug ms = 1 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: debug mgr = 20 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: debug mon = 20 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: debug auth = 20 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: mgr/telemetry/nag = false 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout:[mon] 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: debug ms = 0/0 2026-02-26T03:31:24.105 INFO:teuthology.orchestra.run.trial106.stdout: debug mon = 0/0 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: debug paxos = 0/0 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: debug auth = 20 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: mon data avail warn = 5 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: mon mgr mkfs grace = 240 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: mon osd reporter subtree level = osd 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: mon osd prime pg temp = true 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:31:24.106 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: # 1m isn't quite enough 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: osd pool default crimson = True 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout:[client] 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: rgw cache enabled = true 2026-02-26T03:31:24.107 INFO:teuthology.orchestra.run.trial106.stdout: rgw enable ops log = true 2026-02-26T03:31:24.108 INFO:teuthology.orchestra.run.trial106.stdout: rgw enable usage log = true 2026-02-26T03:31:24.108 INFO:teuthology.orchestra.run.trial106.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:31:24.108 INFO:teuthology.orchestra.run.trial106.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:31:24.108 INFO:teuthology.orchestra.run.trial106.stdout:[mon.a] 2026-02-26T03:31:24.109 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:31:24.109 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:31:24.198 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:31:24.198 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:24.199 DEBUG:teuthology.orchestra.run.trial106:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:31:24.215 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:24.215 DEBUG:teuthology.orchestra.run.trial106:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:31:24.271 INFO:tasks.ceph:Sending monmap to node ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:31:24.272 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:24.272 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:31:24.272 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:31:24.344 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:24.344 DEBUG:teuthology.orchestra.run.trial106:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:31:24.400 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:31:24.400 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:31:24.401 DEBUG:teuthology.orchestra.run.trial106:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:31:24.490 INFO:teuthology.orchestra.run.trial106.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:31:24.493 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:31:24.493 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:31:24.494 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:31:24.535 INFO:teuthology.orchestra.run.trial106.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:31:24.547 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:31:24.547 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial106.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:31:24.547 DEBUG:teuthology.orchestra.run.trial106:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:31:24.611 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:31:24.611 INFO:tasks.ceph:role: osd.0 2026-02-26T03:31:24.611 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:31:24.611 DEBUG:teuthology.orchestra.run.trial106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:31:24.677 INFO:teuthology.orchestra.run.trial106.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:31:24.677 INFO:teuthology.orchestra.run.trial106.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:31:24.677 INFO:teuthology.orchestra.run.trial106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:31:24.678 INFO:teuthology.orchestra.run.trial106.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:31:24.678 INFO:teuthology.orchestra.run.trial106.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:31:24.678 INFO:teuthology.orchestra.run.trial106.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:31:24.678 INFO:teuthology.orchestra.run.trial106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:31:24.678 INFO:teuthology.orchestra.run.trial106.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:31:24.678 INFO:teuthology.orchestra.run.trial106.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:31:24.678 INFO:teuthology.orchestra.run.trial106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:31:25.249 INFO:teuthology.orchestra.run.trial106.stdout:Discarding blocks...Done. 2026-02-26T03:31:25.252 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial106.front.sepia.ceph.com -o noatime 2026-02-26T03:31:25.252 DEBUG:teuthology.orchestra.run.trial106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:31:25.321 DEBUG:teuthology.orchestra.run.trial106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:31:25.388 DEBUG:teuthology.orchestra.run.trial106:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:31:25.453 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:31:25.453 INFO:tasks.ceph:role: osd.1 2026-02-26T03:31:25.453 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:31:25.453 DEBUG:teuthology.orchestra.run.trial106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:31:25.519 INFO:teuthology.orchestra.run.trial106.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:31:25.520 INFO:teuthology.orchestra.run.trial106.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:31:25.521 INFO:teuthology.orchestra.run.trial106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:31:26.078 INFO:teuthology.orchestra.run.trial106.stdout:Discarding blocks...Done. 2026-02-26T03:31:26.081 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial106.front.sepia.ceph.com -o noatime 2026-02-26T03:31:26.081 DEBUG:teuthology.orchestra.run.trial106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:31:26.149 DEBUG:teuthology.orchestra.run.trial106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:31:26.216 DEBUG:teuthology.orchestra.run.trial106:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:31:26.281 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:31:26.281 INFO:tasks.ceph:role: osd.2 2026-02-26T03:31:26.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:31:26.281 DEBUG:teuthology.orchestra.run.trial106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:31:26.347 INFO:teuthology.orchestra.run.trial106.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:31:26.348 INFO:teuthology.orchestra.run.trial106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:31:26.904 INFO:teuthology.orchestra.run.trial106.stdout:Discarding blocks...Done. 2026-02-26T03:31:26.906 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial106.front.sepia.ceph.com -o noatime 2026-02-26T03:31:26.906 DEBUG:teuthology.orchestra.run.trial106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:31:26.974 DEBUG:teuthology.orchestra.run.trial106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:31:27.041 DEBUG:teuthology.orchestra.run.trial106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:31:27.193 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:31:27.251 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:27,251 seastar - Reactor backend: linux-aio 2026-02-26T03:31:27.260 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:27,261 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:27.261 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:27,261 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:27.263 INFO:teuthology.orchestra.run.trial106.stderr:==39908==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:31:27.323 INFO:teuthology.orchestra.run.trial106.stderr:DEBUG 2026-02-26 03:31:27,324 [shard 0:main] seastar - reactor::drain 2026-02-26T03:31:29.379 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:29.379 INFO:teuthology.orchestra.run.trial106.stderr:Suppressions used: 2026-02-26T03:31:29.379 INFO:teuthology.orchestra.run.trial106.stderr: count bytes template 2026-02-26T03:31:29.379 INFO:teuthology.orchestra.run.trial106.stderr: 1 8 MallocExtension::Register 2026-02-26T03:31:29.379 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:29.380 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:31:29.398 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:31:29.398 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:31:29.442 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:31:29.443 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:31:29.449 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:29,449 seastar - Reactor backend: linux-aio 2026-02-26T03:31:29.464 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:29,464 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:29.464 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:29,464 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:29.469 INFO:teuthology.orchestra.run.trial106.stderr:==39906==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:31:29.469 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:29,469 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:31:29.607 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e8f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xfd19a /lib64/libasan.so.8+0xef979 0x31724fc 0x31726d6 0x317298d 0x3172a82 0x6f7048a 0x9d283ce 0x9d285e6 0x9d288b8 0x7be8ba4 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:29.607 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xb9e377 0x9d48623 0x9d48963 0x9d48b2c 0x9d48bbf 0x9d49a04 0x78947dd 0x7beb047 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:29.608 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 34 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x316f517 0x9d43e50 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:29.760 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xad3ac 0xe0b9dd 0x7bea2cf 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:29.760 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:29.760 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15e37 0x9d3f323 0x9d49b10 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:31.887 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:31.888 INFO:teuthology.orchestra.run.trial106.stderr:Suppressions used: 2026-02-26T03:31:31.888 INFO:teuthology.orchestra.run.trial106.stderr: count bytes template 2026-02-26T03:31:31.888 INFO:teuthology.orchestra.run.trial106.stderr: 1 8 MallocExtension::Register 2026-02-26T03:31:31.888 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:31.888 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:31:31.918 DEBUG:teuthology.orchestra.run.trial106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:31:31.944 DEBUG:teuthology.orchestra.run.trial106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:31:32.096 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:31:32.147 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:32,147 seastar - Reactor backend: linux-aio 2026-02-26T03:31:32.158 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:32,158 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:32.158 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:32,158 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:32.160 INFO:teuthology.orchestra.run.trial106.stderr:==39979==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:31:32.219 INFO:teuthology.orchestra.run.trial106.stderr:DEBUG 2026-02-26 03:31:32,219 [shard 0:main] seastar - reactor::drain 2026-02-26T03:31:34.271 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:34.271 INFO:teuthology.orchestra.run.trial106.stderr:Suppressions used: 2026-02-26T03:31:34.271 INFO:teuthology.orchestra.run.trial106.stderr: count bytes template 2026-02-26T03:31:34.271 INFO:teuthology.orchestra.run.trial106.stderr: 1 8 MallocExtension::Register 2026-02-26T03:31:34.272 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:34.272 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:31:34.291 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:31:34.291 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:31:34.332 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:31:34.333 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:31:34.339 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:34,340 seastar - Reactor backend: linux-aio 2026-02-26T03:31:34.347 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:34,347 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:34.347 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:34,347 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:34.352 INFO:teuthology.orchestra.run.trial106.stderr:==39977==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:31:34.352 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:34,352 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:31:34.489 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x361f /lib64/libasan.so.8+0x157c7 0x9d24b21 0x9d25e30 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:34.489 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1fb10 /lib64/libasan.so.8+0x157c7 0x9c51220 0x9c52ef3 0x9c61c89 0x9c6209b 0x9d328ee 0x9d32d22 0x9d330b5 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:34.490 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbf3 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x360a9a3 0x9d234f3 0x9d24b15 0x9d25e30 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:34.635 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x34fd0da 0x35ed768 0x35ed86b 0x36096e8 0x9c63157 0x9c63219 0x9d23007 0x9d2319c 0x9d2352e 0x9d24b15 0x9d25e30 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:34.635 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba35e4 0x9ba325d 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9ba4c54 0x9d1b824 0x9d1b91d 0x7bea265 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:34.635 INFO:teuthology.orchestra.run.trial106.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb61ff44 0xffffffffbb6203b0 0xffffffffbb49525e 0xffffffffbb7e7269 0xffffffffbb7ca7d5 0xffffffffbb7d4bab 0xffffffffbb7d500b 0xffffffffbb4910d2 0xffffffffbc150451 0xffffffffbc200ba2 2026-02-26T03:31:34.635 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b377 /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xb69757 0x1090878 0xf2e0dc 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:36.753 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:36.754 INFO:teuthology.orchestra.run.trial106.stderr:Suppressions used: 2026-02-26T03:31:36.754 INFO:teuthology.orchestra.run.trial106.stderr: count bytes template 2026-02-26T03:31:36.754 INFO:teuthology.orchestra.run.trial106.stderr: 1 8 MallocExtension::Register 2026-02-26T03:31:36.754 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:36.754 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:31:36.783 DEBUG:teuthology.orchestra.run.trial106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:31:36.809 DEBUG:teuthology.orchestra.run.trial106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:31:36.961 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:31:37.021 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:37,021 seastar - Reactor backend: linux-aio 2026-02-26T03:31:37.034 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:37,034 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:37.034 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:37,034 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:37.036 INFO:teuthology.orchestra.run.trial106.stderr:==40044==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:31:37.096 INFO:teuthology.orchestra.run.trial106.stderr:DEBUG 2026-02-26 03:31:37,096 [shard 0:main] seastar - reactor::drain 2026-02-26T03:31:39.150 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:39.150 INFO:teuthology.orchestra.run.trial106.stderr:Suppressions used: 2026-02-26T03:31:39.150 INFO:teuthology.orchestra.run.trial106.stderr: count bytes template 2026-02-26T03:31:39.150 INFO:teuthology.orchestra.run.trial106.stderr: 1 8 MallocExtension::Register 2026-02-26T03:31:39.150 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:39.150 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:31:39.169 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:31:39.170 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:31:39.217 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:31:39.218 INFO:teuthology.orchestra.run.trial106.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:31:39.224 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:39,225 seastar - Reactor backend: linux-aio 2026-02-26T03:31:39.238 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:39,238 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:39.238 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:39,238 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:31:39.242 INFO:teuthology.orchestra.run.trial106.stderr:==40042==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:31:39.242 INFO:teuthology.orchestra.run.trial106.stderr:INFO 2026-02-26 03:31:39,242 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:31:39.274 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0xb75c8b 0x353ed30 0x3570df3 0x35711ca 0x3572686 0x3572881 0x3572a5a 0x35748f1 0x33db40f 0x345a001 0x468369c 0x46ae193 0x46af659 0x46af6de 0x46af7e0 0x46af8d8 0x46afae3 0x46afcc9 0x46afd44 0x46afdbf 0x46afe3a 0x46b07a7 0x46b0b48 0x46b0d73 0x46b0fda 0x46b12a5 0x46b14c3 0x46b157b 0x46b1633 0x46b16ec 0x46b17a4 0x46b186a 0x46b194b 0x311177a 0x3111880 0x3111938 0x31119f0 0x3111aa8 0x3145e83 0x31a5653 0x31a5a14 0x31a5b56 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:31:39.386 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10b696 0xcde7b0 0x9bf024f 0x9d2613f 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:39.386 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe0b7b5 0xe0b9eb 0x7bea0a6 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:39.386 INFO:teuthology.orchestra.run.trial106.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb7fdd1c 0xffffffffbb825a53 0xffffffffbb825c15 0xffffffffbb7ca888 0xffffffffbb7d4bab 0xffffffffbb7d500b 0xffffffffbb4910d2 0xffffffffbc150451 0xffffffffbc200ba2 2026-02-26T03:31:39.386 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xfdf2c /lib64/libasan.so.8+0xeb6b /lib64/libasan.so.8+0xef905 0x9d283db 0x9d285e6 0x9d288b8 0x7bea32f 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:39.531 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x35ed737 0x35ed86b 0x36096e8 0x9c63157 0x9c63219 0x9d23007 0x9d2319c 0x9d2352e 0x9d24b15 0x9d25e30 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:39.531 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcb03 /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c521cc 0x9c5a24f 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:39.531 INFO:teuthology.orchestra.run.trial106.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297b3 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:31:41.656 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:41.656 INFO:teuthology.orchestra.run.trial106.stderr:Suppressions used: 2026-02-26T03:31:41.656 INFO:teuthology.orchestra.run.trial106.stderr: count bytes template 2026-02-26T03:31:41.656 INFO:teuthology.orchestra.run.trial106.stderr: 1 8 MallocExtension::Register 2026-02-26T03:31:41.656 INFO:teuthology.orchestra.run.trial106.stderr:----------------------------------------------------- 2026-02-26T03:31:41.656 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:31:41.686 DEBUG:teuthology.orchestra.run.trial106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:31:41.712 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:31:41.713 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:41.713 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:31:41.777 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:41.777 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:31:41.841 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:41.841 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:31:41.906 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:41.906 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:31:41.970 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:41.970 DEBUG:teuthology.orchestra.run.trial106:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:31:42.026 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:31:42.026 DEBUG:teuthology.orchestra.run.trial106:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:31:42.088 INFO:teuthology.orchestra.run.trial106.stdout:[mgr.x] 2026-02-26T03:31:42.088 INFO:teuthology.orchestra.run.trial106.stdout: key = AQCMvp9pBBlBHRAARGmWH0xdGQo8ESVX0GErsQ== 2026-02-26T03:31:42.088 INFO:teuthology.orchestra.run.trial106.stdout:[osd.0] 2026-02-26T03:31:42.088 INFO:teuthology.orchestra.run.trial106.stdout: key = AQCRvp9pqQvLHxAACfkEw4J9MZwdTjSX04Dz/A== 2026-02-26T03:31:42.089 INFO:teuthology.orchestra.run.trial106.stdout:[osd.1] 2026-02-26T03:31:42.089 INFO:teuthology.orchestra.run.trial106.stdout: key = AQCWvp9pDPLIGBAARQr7gjGCFsXVrWGNfLHFIw== 2026-02-26T03:31:42.089 INFO:teuthology.orchestra.run.trial106.stdout:[osd.2] 2026-02-26T03:31:42.089 INFO:teuthology.orchestra.run.trial106.stdout: key = AQCbvp9pmhGcEhAAWOD1iekaIGtlcY25A+32oQ== 2026-02-26T03:31:42.089 INFO:teuthology.orchestra.run.trial106.stdout:[client.0] 2026-02-26T03:31:42.089 INFO:teuthology.orchestra.run.trial106.stdout: key = AQCMvp9p+vHqHxAAaKpXrlF93h3qZInTBKvb6Q== 2026-02-26T03:31:42.090 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:31:42.181 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:31:42.230 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:31:42.282 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:31:42.331 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:31:42.382 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:31:42.382 DEBUG:teuthology.orchestra.run.trial106:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:31:42.408 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:31:42.495 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.495+0000 7f58d68dec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.495 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.495+0000 7f58d68dec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.503 DEBUG:teuthology.orchestra.run.trial106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:31:42.528 DEBUG:teuthology.orchestra.run.trial106:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:31:42.585 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:31:42.585 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:31:42.585 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:31:42.627 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:31:42.627 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:31:42.628 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:31:42.628 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:31:42.630 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:31:42.630 DEBUG:tasks.ceph:set 0 configs 2026-02-26T03:31:42.631 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph config dump 2026-02-26T03:31:42.689 INFO:tasks.ceph.mon.a.trial106.stderr:2026-02-26T03:31:42.689+0000 7f5b68e86c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.690 INFO:tasks.ceph.mon.a.trial106.stderr:2026-02-26T03:31:42.689+0000 7f5b68e86c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.690 INFO:tasks.ceph.mon.a.trial106.stderr:2026-02-26T03:31:42.689+0000 7f5b68e86c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.693 INFO:tasks.ceph.mgr.x.trial106.stderr:2026-02-26T03:31:42.693+0000 7f46493c3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.707 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.705+0000 7f68214cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.709 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.709+0000 7f68214cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.740 INFO:teuthology.orchestra.run.trial106.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-26T03:31:42.748 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:31:42.748 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:31:42.822 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.819+0000 7fbcfee11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.823 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.823+0000 7fbcfee11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.856 INFO:teuthology.orchestra.run.trial106.stderr:adjusted tunables profile to default 2026-02-26T03:31:42.865 INFO:tasks.ceph:check_enable_crimson: True 2026-02-26T03:31:42.865 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-26T03:31:42.865 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-26T03:31:42.895 INFO:tasks.ceph.mgr.x.trial106.stderr:2026-02-26T03:31:42.895+0000 7f46493c3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.896 INFO:tasks.ceph.mgr.x.trial106.stderr:2026-02-26T03:31:42.895+0000 7f46493c3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.942 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.939+0000 7f46ab7cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.943 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:42.943+0000 7f46ab7cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:42.981 INFO:tasks.ceph.mon.a.trial106.stderr:2026-02-26T03:31:42.981+0000 7f5b605f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-26T03:31:42.982 INFO:tasks.ceph.mon.a.trial106.stderr:Please be aware that this feature is experimental, untested, 2026-02-26T03:31:42.982 INFO:tasks.ceph.mon.a.trial106.stderr:unsupported, and may result in data corruption, data loss, 2026-02-26T03:31:42.982 INFO:tasks.ceph.mon.a.trial106.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-26T03:31:42.982 INFO:tasks.ceph.mon.a.trial106.stderr:feature with important data. 2026-02-26T03:31:42.982 INFO:tasks.ceph.mon.a.trial106.stderr: 2026-02-26T03:31:43.737 INFO:tasks.ceph.mgr.x.trial106.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:31:43.738 INFO:tasks.ceph.mgr.x.trial106.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:31:43.738 INFO:tasks.ceph.mgr.x.trial106.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:31:43.868 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:31:43.868 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:43.869 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:31:43.933 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:43.933 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:31:43.997 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:31:43.997 DEBUG:teuthology.orchestra.run.trial106:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:31:44.061 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph osd new 93dd6b90-3fa5-4805-ab9a-6e85cad63d12 0 2026-02-26T03:31:44.176 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.172+0000 7f6d4b01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.177 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.177+0000 7f6d4b01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.215 INFO:teuthology.orchestra.run.trial106.stdout:0 2026-02-26T03:31:44.224 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph osd new a3cbb47e-2c78-48f6-aa46-a3d5bbd13e46 1 2026-02-26T03:31:44.299 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.296+0000 7f46fd5ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.301 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.301+0000 7f46fd5ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.337 INFO:teuthology.orchestra.run.trial106.stdout:1 2026-02-26T03:31:44.345 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph osd new be341fd4-6b2d-477f-8511-2656df55ac28 2 2026-02-26T03:31:44.419 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.416+0000 7f8338dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.420 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.420+0000 7f8338dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.457 INFO:teuthology.orchestra.run.trial106.stdout:2 2026-02-26T03:31:44.466 DEBUG:teuthology.orchestra.run.trial106:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-26T03:31:44.496 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:31:44.496 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-26T03:31:44.538 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:31:44.538 DEBUG:teuthology.orchestra.run.trial106:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-26T03:31:44.567 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:31:44.568 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-26T03:31:44.611 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:31:44.611 DEBUG:teuthology.orchestra.run.trial106:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-26T03:31:44.640 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:31:44.640 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-26T03:31:44.682 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:31:44.683 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:31:44.760 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.758+0000 7f5754ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.761 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:44.761+0000 7f5754ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:44.793 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:31:44.793 INFO:teuthology.orchestra.run.trial106.stdout:{"epoch":6,"fsid":"e84db318-d879-4a23-9cae-25a52f67fed2","created":"2026-02-26T03:31:42.704444+0000","modified":"2026-02-26T03:31:44.457224+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:31:44.457224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"93dd6b90-3fa5-4805-ab9a-6e85cad63d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3cbb47e-2c78-48f6-aa46-a3d5bbd13e46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"be341fd4-6b2d-477f-8511-2656df55ac28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:31:44.803 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:31:44.803 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:31:45.105 DEBUG:teuthology.orchestra.run.trial106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:31:45.174 INFO:teuthology.misc.health.trial106.stderr:2026-02-26T03:31:45.171+0000 7f78a6ac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:45.175 INFO:teuthology.misc.health.trial106.stderr:2026-02-26T03:31:45.175+0000 7f78a6ac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:45.206 INFO:teuthology.misc.health.trial106.stdout: 2026-02-26T03:31:45.206 INFO:teuthology.misc.health.trial106.stdout:{"epoch":6,"fsid":"e84db318-d879-4a23-9cae-25a52f67fed2","created":"2026-02-26T03:31:42.704444+0000","modified":"2026-02-26T03:31:44.457224+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:31:44.457224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"93dd6b90-3fa5-4805-ab9a-6e85cad63d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3cbb47e-2c78-48f6-aa46-a3d5bbd13e46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"be341fd4-6b2d-477f-8511-2656df55ac28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:31:45.214 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:31:46.150 INFO:tasks.ceph.mgr.x.trial106.stderr:2026-02-26T03:31:46.150+0000 7f4564333640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:31:51.516 DEBUG:teuthology.orchestra.run.trial106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:31:51.585 INFO:teuthology.misc.health.trial106.stderr:2026-02-26T03:31:51.581+0000 7ff29c70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:51.586 INFO:teuthology.misc.health.trial106.stderr:2026-02-26T03:31:51.585+0000 7ff29c70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:51.698 INFO:teuthology.misc.health.trial106.stdout: 2026-02-26T03:31:51.698 INFO:teuthology.misc.health.trial106.stdout:{"epoch":8,"fsid":"e84db318-d879-4a23-9cae-25a52f67fed2","created":"2026-02-26T03:31:42.704444+0000","modified":"2026-02-26T03:31:51.600420+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:31:44.457224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"93dd6b90-3fa5-4805-ab9a-6e85cad63d12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a3cbb47e-2c78-48f6-aa46-a3d5bbd13e46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"be341fd4-6b2d-477f-8511-2656df55ac28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:31:51.707 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:31:58.010 DEBUG:teuthology.orchestra.run.trial106:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:31:58.077 INFO:teuthology.misc.health.trial106.stderr:2026-02-26T03:31:58.074+0000 7f42a4411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:58.078 INFO:teuthology.misc.health.trial106.stderr:2026-02-26T03:31:58.077+0000 7f42a4411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:58.189 INFO:teuthology.misc.health.trial106.stdout: 2026-02-26T03:31:58.189 INFO:teuthology.misc.health.trial106.stdout:{"epoch":13,"fsid":"e84db318-d879-4a23-9cae-25a52f67fed2","created":"2026-02-26T03:31:42.704444+0000","modified":"2026-02-26T03:31:57.613935+0000","last_up_change":"2026-02-26T03:31:53.605503+0000","last_in_change":"2026-02-26T03:31:44.457224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:31:55.159637+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93dd6b90-3fa5-4805-ab9a-6e85cad63d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6803","nonce":1940289837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6802","nonce":1940289837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6807","nonce":1940289837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6810","nonce":1940289837}]},"public_addr":"10.20.193.106:6803/1940289837","cluster_addr":"10.20.193.106:6802/1940289837","heartbeat_back_addr":"10.20.193.106:6807/1940289837","heartbeat_front_addr":"10.20.193.106:6810/1940289837","state":["exists","up"]},{"osd":1,"uuid":"a3cbb47e-2c78-48f6-aa46-a3d5bbd13e46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6809","nonce":3722933593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6808","nonce":3722933593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6804","nonce":3722933593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6811","nonce":3722933593}]},"public_addr":"10.20.193.106:6809/3722933593","cluster_addr":"10.20.193.106:6808/3722933593","heartbeat_back_addr":"10.20.193.106:6804/3722933593","heartbeat_front_addr":"10.20.193.106:6811/3722933593","state":["exists","up"]},{"osd":2,"uuid":"be341fd4-6b2d-477f-8511-2656df55ac28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6806","nonce":2502740801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6805","nonce":2502740801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6816","nonce":2502740801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6815","nonce":2502740801}]},"public_addr":"10.20.193.106:6806/2502740801","cluster_addr":"10.20.193.106:6805/2502740801","heartbeat_back_addr":"10.20.193.106:6816/2502740801","heartbeat_front_addr":"10.20.193.106:6815/2502740801","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:31:58.196 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:31:58.196 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:31:58.197 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:31:58.272 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:58.268+0000 7f7ceeab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:58.273 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:58.272+0000 7f7ceeab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:58.947 INFO:teuthology.orchestra.run.trial106.stderr:pool 'rbd' created 2026-02-26T03:31:58.962 DEBUG:teuthology.orchestra.run.trial106:> rbd --cluster ceph pool init rbd 2026-02-26T03:31:59.007 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:59.006+0000 7f5dddfee240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:59.008 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:59.006+0000 7f5dddfee240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:31:59.009 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:31:59.008+0000 7f5dddfee240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:01.988 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:32:01.989 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-26T03:32:01.990 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:32:02.066 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.063+0000 7f6b87f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.067 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.066+0000 7f6b87f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.177 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:32:02.186 INFO:teuthology.orchestra.run.trial106.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-26T03:32:02.186 INFO:tasks.ceph_manager:config epoch is 1 2026-02-26T03:32:02.187 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:32:02.187 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:32:02.187 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:32:02.265 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.262+0000 7f26c4a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.266 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.265+0000 7f26c4a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.410 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:32:02.418 INFO:teuthology.orchestra.run.trial106.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6800","nonce":4131200032},{"type":"v1","addr":"10.20.193.106:6801","nonce":4131200032}]},"active_addr":"10.20.193.106:6801/4131200032","active_change":"2026-02-26T03:31:45.144694+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.106:0","nonce":1615404001}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.106:0","nonce":1862081870}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.106:0","nonce":2161144733}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.106:0","nonce":1979495294}]}]} 2026-02-26T03:32:02.422 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:32:02.423 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:32:02.423 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:32:02.500 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.497+0000 7fe67f2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.501 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.500+0000 7fe67f2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.613 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:32:02.613 INFO:teuthology.orchestra.run.trial106.stdout:{"epoch":17,"fsid":"e84db318-d879-4a23-9cae-25a52f67fed2","created":"2026-02-26T03:31:42.704444+0000","modified":"2026-02-26T03:32:01.959622+0000","last_up_change":"2026-02-26T03:31:53.605503+0000","last_in_change":"2026-02-26T03:31:44.457224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:31:55.159637+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:31:58.387528+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93dd6b90-3fa5-4805-ab9a-6e85cad63d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6803","nonce":1940289837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6802","nonce":1940289837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6807","nonce":1940289837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6810","nonce":1940289837}]},"public_addr":"10.20.193.106:6803/1940289837","cluster_addr":"10.20.193.106:6802/1940289837","heartbeat_back_addr":"10.20.193.106:6807/1940289837","heartbeat_front_addr":"10.20.193.106:6810/1940289837","state":["exists","up"]},{"osd":1,"uuid":"a3cbb47e-2c78-48f6-aa46-a3d5bbd13e46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6809","nonce":3722933593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6808","nonce":3722933593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6804","nonce":3722933593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6811","nonce":3722933593}]},"public_addr":"10.20.193.106:6809/3722933593","cluster_addr":"10.20.193.106:6808/3722933593","heartbeat_back_addr":"10.20.193.106:6804/3722933593","heartbeat_front_addr":"10.20.193.106:6811/3722933593","state":["exists","up"]},{"osd":2,"uuid":"be341fd4-6b2d-477f-8511-2656df55ac28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6806","nonce":2502740801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6805","nonce":2502740801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6816","nonce":2502740801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6815","nonce":2502740801}]},"public_addr":"10.20.193.106:6806/2502740801","cluster_addr":"10.20.193.106:6805/2502740801","heartbeat_back_addr":"10.20.193.106:6816/2502740801","heartbeat_front_addr":"10.20.193.106:6815/2502740801","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:32:02.623 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:32:02.623 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:32:02.701 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.698+0000 7f0ec07dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.702 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.701+0000 7f0ec07dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.812 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:32:02.812 INFO:teuthology.orchestra.run.trial106.stdout:{"epoch":17,"fsid":"e84db318-d879-4a23-9cae-25a52f67fed2","created":"2026-02-26T03:31:42.704444+0000","modified":"2026-02-26T03:32:01.959622+0000","last_up_change":"2026-02-26T03:31:53.605503+0000","last_in_change":"2026-02-26T03:31:44.457224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:31:55.159637+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:31:58.387528+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93dd6b90-3fa5-4805-ab9a-6e85cad63d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6803","nonce":1940289837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6802","nonce":1940289837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6807","nonce":1940289837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6810","nonce":1940289837}]},"public_addr":"10.20.193.106:6803/1940289837","cluster_addr":"10.20.193.106:6802/1940289837","heartbeat_back_addr":"10.20.193.106:6807/1940289837","heartbeat_front_addr":"10.20.193.106:6810/1940289837","state":["exists","up"]},{"osd":1,"uuid":"a3cbb47e-2c78-48f6-aa46-a3d5bbd13e46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6809","nonce":3722933593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6808","nonce":3722933593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6804","nonce":3722933593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6811","nonce":3722933593}]},"public_addr":"10.20.193.106:6809/3722933593","cluster_addr":"10.20.193.106:6808/3722933593","heartbeat_back_addr":"10.20.193.106:6804/3722933593","heartbeat_front_addr":"10.20.193.106:6811/3722933593","state":["exists","up"]},{"osd":2,"uuid":"be341fd4-6b2d-477f-8511-2656df55ac28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6806","nonce":2502740801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6805","nonce":2502740801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6816","nonce":2502740801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.106:6815","nonce":2502740801}]},"public_addr":"10.20.193.106:6806/2502740801","cluster_addr":"10.20.193.106:6805/2502740801","heartbeat_back_addr":"10.20.193.106:6816/2502740801","heartbeat_front_addr":"10.20.193.106:6815/2502740801","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:32:02.823 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:32:02.824 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:32:02.824 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:32:02.905 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.901+0000 7ff792bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.905 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.902+0000 7f2dd50c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.906 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.902+0000 7f1938f7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.906 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.905+0000 7ff792bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.907 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.906+0000 7f2dd50c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.907 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:02.906+0000 7f1938f7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:02.920 INFO:teuthology.orchestra.run.trial106.stdout:42949672969 2026-02-26T03:32:02.921 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:32:02.922 INFO:teuthology.orchestra.run.trial106.stdout:38654705674 2026-02-26T03:32:02.922 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:32:02.923 INFO:teuthology.orchestra.run.trial106.stdout:42949672969 2026-02-26T03:32:02.924 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:32:03.042 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.038+0000 7f1f89a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.043 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.039+0000 7f761cc3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.044 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.043+0000 7f1f89a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.045 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.044+0000 7f761cc3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.083 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.079+0000 7f175219c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.084 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.083+0000 7f175219c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.155 INFO:teuthology.orchestra.run.trial106.stdout:42949672969 2026-02-26T03:32:03.162 INFO:teuthology.orchestra.run.trial106.stdout:38654705674 2026-02-26T03:32:03.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.1 2026-02-26T03:32:03.167 DEBUG:teuthology.parallel:result is None 2026-02-26T03:32:03.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2026-02-26T03:32:03.174 DEBUG:teuthology.parallel:result is None 2026-02-26T03:32:03.197 INFO:teuthology.orchestra.run.trial106.stdout:42949672969 2026-02-26T03:32:03.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-26T03:32:03.207 DEBUG:teuthology.parallel:result is None 2026-02-26T03:32:03.207 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:32:03.208 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:32:03.323 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.321+0000 7eff887dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.325 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.324+0000 7eff887dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.436 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:32:03.436 INFO:teuthology.orchestra.run.trial106.stderr:dumped all 2026-02-26T03:32:03.445 INFO:teuthology.orchestra.run.trial106.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:32:03.148748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.233869"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.976454+0000","last_change":"2026-02-26T03:32:01.979210+0000","last_active":"2026-02-26T03:32:01.976454+0000","last_peered":"2026-02-26T03:32:01.976454+0000","last_clean":"2026-02-26T03:32:01.976454+0000","last_became_active":"2026-02-26T03:31:59.982123+0000","last_became_peered":"2026-02-26T03:31:59.982123+0000","last_unstale":"2026-02-26T03:32:01.976454+0000","last_undegraded":"2026-02-26T03:32:01.976454+0000","last_fullsized":"2026-02-26T03:32:01.976454+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.977268+0000","last_change":"2026-02-26T03:32:01.980115+0000","last_active":"2026-02-26T03:32:01.977268+0000","last_peered":"2026-02-26T03:32:01.977268+0000","last_clean":"2026-02-26T03:32:01.977268+0000","last_became_active":"2026-02-26T03:31:59.981741+0000","last_became_peered":"2026-02-26T03:31:59.981741+0000","last_unstale":"2026-02-26T03:32:01.977268+0000","last_undegraded":"2026-02-26T03:32:01.977268+0000","last_fullsized":"2026-02-26T03:32:01.977268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.977274+0000","last_change":"2026-02-26T03:32:01.980557+0000","last_active":"2026-02-26T03:32:01.977274+0000","last_peered":"2026-02-26T03:32:01.977274+0000","last_clean":"2026-02-26T03:32:01.977274+0000","last_became_active":"2026-02-26T03:31:59.991158+0000","last_became_peered":"2026-02-26T03:31:59.991158+0000","last_unstale":"2026-02-26T03:32:01.977274+0000","last_undegraded":"2026-02-26T03:32:01.977274+0000","last_fullsized":"2026-02-26T03:32:01.977274+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.977306+0000","last_change":"2026-02-26T03:32:01.978980+0000","last_active":"2026-02-26T03:32:01.977306+0000","last_peered":"2026-02-26T03:32:01.977306+0000","last_clean":"2026-02-26T03:32:01.977306+0000","last_became_active":"2026-02-26T03:31:59.984832+0000","last_became_peered":"2026-02-26T03:31:59.984832+0000","last_unstale":"2026-02-26T03:32:01.977306+0000","last_undegraded":"2026-02-26T03:32:01.977306+0000","last_fullsized":"2026-02-26T03:32:01.977306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.974755+0000","last_change":"2026-02-26T03:32:01.974755+0000","last_active":"2026-02-26T03:32:01.974755+0000","last_peered":"2026-02-26T03:32:01.974755+0000","last_clean":"2026-02-26T03:32:01.974755+0000","last_became_active":"2026-02-26T03:31:59.992998+0000","last_became_peered":"2026-02-26T03:31:59.992998+0000","last_unstale":"2026-02-26T03:32:01.974755+0000","last_undegraded":"2026-02-26T03:32:01.974755+0000","last_fullsized":"2026-02-26T03:32:01.974755+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.974187+0000","last_change":"2026-02-26T03:32:01.975936+0000","last_active":"2026-02-26T03:32:01.974187+0000","last_peered":"2026-02-26T03:32:01.974187+0000","last_clean":"2026-02-26T03:32:01.974187+0000","last_became_active":"2026-02-26T03:31:59.997745+0000","last_became_peered":"2026-02-26T03:31:59.997745+0000","last_unstale":"2026-02-26T03:32:01.974187+0000","last_undegraded":"2026-02-26T03:32:01.974187+0000","last_fullsized":"2026-02-26T03:32:01.974187+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.972410+0000","last_change":"2026-02-26T03:32:01.974777+0000","last_active":"2026-02-26T03:32:01.972410+0000","last_peered":"2026-02-26T03:32:01.972410+0000","last_clean":"2026-02-26T03:32:01.972410+0000","last_became_active":"2026-02-26T03:31:59.988546+0000","last_became_peered":"2026-02-26T03:31:59.988546+0000","last_unstale":"2026-02-26T03:32:01.972410+0000","last_undegraded":"2026-02-26T03:32:01.972410+0000","last_fullsized":"2026-02-26T03:32:01.972410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.978033+0000","last_change":"2026-02-26T03:32:01.980346+0000","last_active":"2026-02-26T03:32:01.978033+0000","last_peered":"2026-02-26T03:32:01.978033+0000","last_clean":"2026-02-26T03:32:01.978033+0000","last_became_active":"2026-02-26T03:31:59.980407+0000","last_became_peered":"2026-02-26T03:31:59.980407+0000","last_unstale":"2026-02-26T03:32:01.978033+0000","last_undegraded":"2026-02-26T03:32:01.978033+0000","last_fullsized":"2026-02-26T03:32:01.978033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.976212+0000","last_change":"2026-02-26T03:31:56.647250+0000","last_active":"2026-02-26T03:32:01.976212+0000","last_peered":"2026-02-26T03:32:01.976212+0000","last_clean":"2026-02-26T03:32:01.976212+0000","last_became_active":"2026-02-26T03:31:56.640103+0000","last_became_peered":"2026-02-26T03:31:56.640103+0000","last_unstale":"2026-02-26T03:32:01.976212+0000","last_undegraded":"2026-02-26T03:32:01.976212+0000","last_fullsized":"2026-02-26T03:32:01.976212+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:55.607003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:55.607003+0000","last_clean_scrub_stamp":"2026-02-26T03:31:55.607003+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:32:03.446 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:32:03.524 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.521+0000 7fa19affd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.525 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.524+0000 7fa19affd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.635 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:32:03.636 INFO:teuthology.orchestra.run.trial106.stderr:dumped all 2026-02-26T03:32:03.646 INFO:teuthology.orchestra.run.trial106.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:32:03.148748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.233869"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.976454+0000","last_change":"2026-02-26T03:32:01.979210+0000","last_active":"2026-02-26T03:32:01.976454+0000","last_peered":"2026-02-26T03:32:01.976454+0000","last_clean":"2026-02-26T03:32:01.976454+0000","last_became_active":"2026-02-26T03:31:59.982123+0000","last_became_peered":"2026-02-26T03:31:59.982123+0000","last_unstale":"2026-02-26T03:32:01.976454+0000","last_undegraded":"2026-02-26T03:32:01.976454+0000","last_fullsized":"2026-02-26T03:32:01.976454+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.977268+0000","last_change":"2026-02-26T03:32:01.980115+0000","last_active":"2026-02-26T03:32:01.977268+0000","last_peered":"2026-02-26T03:32:01.977268+0000","last_clean":"2026-02-26T03:32:01.977268+0000","last_became_active":"2026-02-26T03:31:59.981741+0000","last_became_peered":"2026-02-26T03:31:59.981741+0000","last_unstale":"2026-02-26T03:32:01.977268+0000","last_undegraded":"2026-02-26T03:32:01.977268+0000","last_fullsized":"2026-02-26T03:32:01.977268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.977274+0000","last_change":"2026-02-26T03:32:01.980557+0000","last_active":"2026-02-26T03:32:01.977274+0000","last_peered":"2026-02-26T03:32:01.977274+0000","last_clean":"2026-02-26T03:32:01.977274+0000","last_became_active":"2026-02-26T03:31:59.991158+0000","last_became_peered":"2026-02-26T03:31:59.991158+0000","last_unstale":"2026-02-26T03:32:01.977274+0000","last_undegraded":"2026-02-26T03:32:01.977274+0000","last_fullsized":"2026-02-26T03:32:01.977274+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.977306+0000","last_change":"2026-02-26T03:32:01.978980+0000","last_active":"2026-02-26T03:32:01.977306+0000","last_peered":"2026-02-26T03:32:01.977306+0000","last_clean":"2026-02-26T03:32:01.977306+0000","last_became_active":"2026-02-26T03:31:59.984832+0000","last_became_peered":"2026-02-26T03:31:59.984832+0000","last_unstale":"2026-02-26T03:32:01.977306+0000","last_undegraded":"2026-02-26T03:32:01.977306+0000","last_fullsized":"2026-02-26T03:32:01.977306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.974755+0000","last_change":"2026-02-26T03:32:01.974755+0000","last_active":"2026-02-26T03:32:01.974755+0000","last_peered":"2026-02-26T03:32:01.974755+0000","last_clean":"2026-02-26T03:32:01.974755+0000","last_became_active":"2026-02-26T03:31:59.992998+0000","last_became_peered":"2026-02-26T03:31:59.992998+0000","last_unstale":"2026-02-26T03:32:01.974755+0000","last_undegraded":"2026-02-26T03:32:01.974755+0000","last_fullsized":"2026-02-26T03:32:01.974755+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.974187+0000","last_change":"2026-02-26T03:32:01.975936+0000","last_active":"2026-02-26T03:32:01.974187+0000","last_peered":"2026-02-26T03:32:01.974187+0000","last_clean":"2026-02-26T03:32:01.974187+0000","last_became_active":"2026-02-26T03:31:59.997745+0000","last_became_peered":"2026-02-26T03:31:59.997745+0000","last_unstale":"2026-02-26T03:32:01.974187+0000","last_undegraded":"2026-02-26T03:32:01.974187+0000","last_fullsized":"2026-02-26T03:32:01.974187+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.972410+0000","last_change":"2026-02-26T03:32:01.974777+0000","last_active":"2026-02-26T03:32:01.972410+0000","last_peered":"2026-02-26T03:32:01.972410+0000","last_clean":"2026-02-26T03:32:01.972410+0000","last_became_active":"2026-02-26T03:31:59.988546+0000","last_became_peered":"2026-02-26T03:31:59.988546+0000","last_unstale":"2026-02-26T03:32:01.972410+0000","last_undegraded":"2026-02-26T03:32:01.972410+0000","last_fullsized":"2026-02-26T03:32:01.972410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.978033+0000","last_change":"2026-02-26T03:32:01.980346+0000","last_active":"2026-02-26T03:32:01.978033+0000","last_peered":"2026-02-26T03:32:01.978033+0000","last_clean":"2026-02-26T03:32:01.978033+0000","last_became_active":"2026-02-26T03:31:59.980407+0000","last_became_peered":"2026-02-26T03:31:59.980407+0000","last_unstale":"2026-02-26T03:32:01.978033+0000","last_undegraded":"2026-02-26T03:32:01.978033+0000","last_fullsized":"2026-02-26T03:32:01.978033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:58.946170+0000","last_clean_scrub_stamp":"2026-02-26T03:31:58.946170+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:32:01.976212+0000","last_change":"2026-02-26T03:31:56.647250+0000","last_active":"2026-02-26T03:32:01.976212+0000","last_peered":"2026-02-26T03:32:01.976212+0000","last_clean":"2026-02-26T03:32:01.976212+0000","last_became_active":"2026-02-26T03:31:56.640103+0000","last_became_peered":"2026-02-26T03:31:56.640103+0000","last_unstale":"2026-02-26T03:32:01.976212+0000","last_undegraded":"2026-02-26T03:32:01.976212+0000","last_fullsized":"2026-02-26T03:32:01.976212+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:31:55.607003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:31:55.607003+0000","last_clean_scrub_stamp":"2026-02-26T03:31:55.607003+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:32:03.647 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:32:03.648 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:32:03.648 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:32:03.648 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:32:03.725 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.722+0000 7f7f2319c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.726 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:03.725+0000 7f7f2319c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:03.861 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-26T03:32:03.861 INFO:teuthology.orchestra.run.trial106.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:32:03.871 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:32:03.871 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-26T03:32:04.066 DEBUG:teuthology.orchestra.run.trial106:> echo 'StrictHostKeyChecking no 2026-02-26T03:32:04.066 DEBUG:teuthology.orchestra.run.trial106:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-26T03:32:04.082 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCmYJsDmkoQVUObOTsg/AkJjyl0I4M1FCBKCwOMJ2D66lNDXgdYg480FnPkm4pE0KARSxjUf/BoybdjQtNYwdVKiRw65ImbaSSkO8EYbBiwBf+r2Cbu1ZQrIjlVdiM1/stBbfcrj960koRZVDFNiwP3LuR2kO+6QFJhO6uvYa4HshafAwApjGww5ZD+7D6V5wQiUGhk3G8glRfKnzCIZ6JfPDHRK+pLQWUrJMLgBMioTtLx7oxy/0CCCponZuhsnpkl3D+573u/QBuPUCX4YkcRKj4xY+RIoE1EnOOGyIrJ6CwhdG52xbfvLxEADHioswqdiXN5hwCTWzmwoIXRvWP3 2026-02-26T03:32:04.082 INFO:teuthology.task.ssh_keys:pushing keys to trial106.front.sepia.ceph.com for ubuntu 2026-02-26T03:32:04.082 DEBUG:teuthology.orchestra.run.trial106:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:32:04.139 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:32:04.139 DEBUG:teuthology.orchestra.run.trial106:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-26T03:32:04.139 DEBUG:teuthology.orchestra.run.trial106:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-26T03:32:04.195 DEBUG:teuthology.orchestra.run.trial106:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:32:04.252 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:32:04.252 DEBUG:teuthology.orchestra.run.trial106:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:32:04.308 DEBUG:teuthology.orchestra.run.trial106:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:32:04:308144 2026-02-26T03:32:04.365 INFO:teuthology.orchestra.run.trial106.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-26_03:32:04:308144' 2026-02-26T03:32:04.365 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-26T03:32:04.365 DEBUG:teuthology.orchestra.run.trial106:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-26T03:32:04.422 INFO:teuthology.run_tasks:Running task radosbench... 2026-02-26T03:32:04.448 INFO:tasks.radosbench:Beginning radosbench... 2026-02-26T03:32:04.448 INFO:tasks.radosbench:omap writes 2026-02-26T03:32:04.448 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-26T03:32:04.449 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-26T03:32:04.540 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:04.535+0000 7f080e61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:04.541 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:04.540+0000 7f080e61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:04.969 INFO:teuthology.orchestra.run.trial106.stderr:pool 'unique_pool_0' created 2026-02-26T03:32:04.984 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-26T03:32:05.071 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:05.065+0000 7f20af5c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:05.072 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:32:05.071+0000 7f20af5c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:05.973 INFO:teuthology.orchestra.run.trial106.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-26T03:32:06.992 DEBUG:teuthology.orchestra.run.trial106:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -t 16 -b 4096 --object-size 4096 -p unique_pool_0 bench 300 write --write-omap' 2026-02-26T03:32:06.995 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2026-02-26T03:32:07.010 INFO:tasks.radosbench:joining radosbench (timing out after 9300s) 2026-02-26T03:32:07.010 INFO:teuthology.orchestra.run:waiting for 9300 2026-02-26T03:32:07.040 INFO:tasks.radosbench.radosbench.0.trial106.stdout:hints = 1 2026-02-26T03:32:07.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 300 seconds or 0 objects 2026-02-26T03:32:07.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Object prefix: benchmark_data_trial106_42438 2026-02-26T03:32:07.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:32:07.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 0 0 0 0 0 0 - 0 2026-02-26T03:32:08.040 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 1 16 329 313 1.22254 1.22266 0.0476265 0.0491648 2026-02-26T03:32:09.040 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 2 16 609 593 1.15811 1.09375 0.0187998 0.0516284 2026-02-26T03:32:10.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 3 16 896 880 1.14574 1.12109 0.0374285 0.0529767 2026-02-26T03:32:11.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 4 16 1182 1166 1.13858 1.11719 0.0262855 0.0541304 2026-02-26T03:32:12.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 5 16 1498 1482 1.15772 1.23438 0.0212117 0.053576 2026-02-26T03:32:13.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 6 16 1806 1790 1.16527 1.20312 0.0269994 0.0531992 2026-02-26T03:32:14.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 7 16 2089 2073 1.15672 1.10547 0.0571555 0.053531 2026-02-26T03:32:15.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 8 16 2422 2406 1.17472 1.30078 0.0678773 0.0529082 2026-02-26T03:32:16.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 9 16 2751 2735 1.18698 1.28516 0.0879764 0.0525074 2026-02-26T03:32:17.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 10 16 3082 3066 1.19757 1.29297 0.0191518 0.0520255 2026-02-26T03:32:18.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 11 16 3431 3415 1.21262 1.36328 0.0619218 0.0514145 2026-02-26T03:32:19.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 12 16 3789 3773 1.2281 1.39844 0.0251392 0.0507644 2026-02-26T03:32:20.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 13 16 4123 4107 1.23398 1.30469 0.043765 0.0504999 2026-02-26T03:32:21.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 14 16 4452 4436 1.23763 1.28516 0.0258451 0.0504185 2026-02-26T03:32:22.041 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 15 16 4800 4784 1.24574 1.35938 0.0449025 0.0500225 2026-02-26T03:32:23.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 16 16 5148 5132 1.25283 1.35938 0.0380223 0.0497971 2026-02-26T03:32:24.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 17 16 5489 5473 1.25749 1.33203 0.0356473 0.0495615 2026-02-26T03:32:25.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 18 16 5848 5832 1.26553 1.40234 0.0351837 0.0492977 2026-02-26T03:32:26.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 19 16 6199 6183 1.27108 1.37109 0.053486 0.0490897 2026-02-26T03:32:27.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:32:27.042328+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0489881 2026-02-26T03:32:27.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:32:27.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 20 16 6540 6524 1.27412 1.33203 0.0577609 0.0489881 2026-02-26T03:32:28.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 21 16 6895 6879 1.27948 1.38672 0.0531785 0.0487775 2026-02-26T03:32:29.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 22 16 7234 7218 1.28151 1.32422 0.0763637 0.0486704 2026-02-26T03:32:30.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 23 16 7576 7560 1.28387 1.33594 0.0417374 0.0486039 2026-02-26T03:32:31.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 24 16 7910 7894 1.28473 1.30469 0.0421323 0.0485709 2026-02-26T03:32:32.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 25 16 8265 8249 1.28881 1.38672 0.0282151 0.0484402 2026-02-26T03:32:33.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 26 16 8616 8600 1.29197 1.37109 0.135453 0.0483013 2026-02-26T03:32:34.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 27 16 8959 8943 1.29374 1.33984 0.0324135 0.0482483 2026-02-26T03:32:35.042 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 28 16 9299 9283 1.29496 1.32812 0.0252033 0.0482029 2026-02-26T03:32:36.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 29 16 9647 9631 1.29718 1.35938 0.0259507 0.0481376 2026-02-26T03:32:37.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 30 16 9974 9958 1.29651 1.27734 0.0489794 0.0481556 2026-02-26T03:32:38.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 31 16 10316 10300 1.29778 1.33594 0.0755547 0.0481141 2026-02-26T03:32:39.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 32 16 10656 10640 1.29873 1.32812 0.0671427 0.0480823 2026-02-26T03:32:40.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 33 16 10965 10949 1.29594 1.20703 0.0954501 0.0481875 2026-02-26T03:32:41.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 34 16 11314 11298 1.29792 1.36328 0.0554458 0.0481294 2026-02-26T03:32:42.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 35 16 11650 11634 1.29833 1.3125 0.0693161 0.0480851 2026-02-26T03:32:43.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 36 16 11993 11977 1.29948 1.33984 0.0776143 0.0480509 2026-02-26T03:32:44.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 37 16 12319 12303 1.29878 1.27344 0.0359067 0.0480705 2026-02-26T03:32:45.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 38 16 12651 12635 1.29873 1.29688 0.043207 0.0480833 2026-02-26T03:32:46.043 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 39 16 12983 12967 1.29868 1.29688 0.065113 0.0480912 2026-02-26T03:32:47.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:32:47.043967+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0480848 2026-02-26T03:32:47.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:32:47.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 40 16 13313 13297 1.29843 1.28906 0.0412711 0.0480848 2026-02-26T03:32:48.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 41 16 13624 13608 1.29639 1.21484 0.0190952 0.0481461 2026-02-26T03:32:49.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 42 16 13935 13919 1.29445 1.21484 0.0378355 0.0482546 2026-02-26T03:32:50.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 43 16 14261 14245 1.29396 1.27344 0.0453186 0.0482651 2026-02-26T03:32:51.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 44 16 14579 14563 1.29278 1.24219 0.0399519 0.0483017 2026-02-26T03:32:52.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 45 16 14895 14879 1.29148 1.23438 0.0522122 0.0483613 2026-02-26T03:32:53.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 46 16 15218 15202 1.29083 1.26172 0.0401606 0.0483841 2026-02-26T03:32:54.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 47 16 15516 15500 1.28813 1.16406 0.0399325 0.0484618 2026-02-26T03:32:55.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 48 16 15831 15815 1.28693 1.23047 0.0548705 0.0485354 2026-02-26T03:32:56.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 49 16 16130 16114 1.2845 1.16797 0.0903181 0.0486196 2026-02-26T03:32:57.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 50 16 16451 16435 1.28388 1.25391 0.0783678 0.0486448 2026-02-26T03:32:58.044 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 51 16 16751 16735 1.28168 1.17188 0.0335824 0.0487322 2026-02-26T03:32:59.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 52 16 17065 17049 1.28062 1.22656 0.0164622 0.0487685 2026-02-26T03:33:00.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 53 16 17386 17370 1.28012 1.25391 0.0664746 0.0488 2026-02-26T03:33:01.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 54 16 17689 17673 1.27833 1.18359 0.0529649 0.0488527 2026-02-26T03:33:02.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 55 16 17993 17977 1.27667 1.1875 0.034675 0.0489234 2026-02-26T03:33:03.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 56 16 18285 18269 1.27424 1.14062 0.0351498 0.0490209 2026-02-26T03:33:04.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 57 16 18567 18551 1.27121 1.10156 0.0670133 0.0491386 2026-02-26T03:33:05.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 58 16 18860 18844 1.26903 1.14453 0.0392682 0.0492195 2026-02-26T03:33:06.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 59 16 19179 19163 1.26864 1.24609 0.0500466 0.0492397 2026-02-26T03:33:07.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:33:07.045559+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0492672 2026-02-26T03:33:07.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:33:07.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 60 16 19491 19475 1.2678 1.21875 0.0350621 0.0492672 2026-02-26T03:33:08.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 61 16 19792 19776 1.26629 1.17578 0.0689669 0.0493233 2026-02-26T03:33:09.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 62 16 20094 20078 1.26489 1.17969 0.0327405 0.0493844 2026-02-26T03:33:10.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 63 16 20382 20366 1.26267 1.125 0.0959685 0.0494743 2026-02-26T03:33:11.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 64 16 20684 20668 1.26137 1.17969 0.105169 0.0495196 2026-02-26T03:33:12.045 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 65 16 20986 20970 1.26012 1.17969 0.0466038 0.0495705 2026-02-26T03:33:13.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 66 16 21288 21272 1.2589 1.17969 0.068067 0.049623 2026-02-26T03:33:14.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 67 16 21564 21548 1.2562 1.07812 0.0461679 0.0497295 2026-02-26T03:33:15.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 68 16 21856 21840 1.2545 1.14062 0.0351576 0.0497921 2026-02-26T03:33:16.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 69 16 22150 22134 1.25296 1.14844 0.0443447 0.0498526 2026-02-26T03:33:17.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 70 16 22442 22426 1.25135 1.14062 0.0704255 0.0499243 2026-02-26T03:33:18.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 71 16 22729 22713 1.24952 1.12109 0.132653 0.0499862 2026-02-26T03:33:19.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 72 16 23000 22984 1.24686 1.05859 0.0624859 0.0500964 2026-02-26T03:33:20.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 73 16 23295 23279 1.24557 1.15234 0.0272391 0.0501536 2026-02-26T03:33:21.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 74 16 23571 23555 1.2433 1.07812 0.159181 0.0502287 2026-02-26T03:33:22.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 75 16 23862 23846 1.24188 1.13672 0.0249849 0.0503008 2026-02-26T03:33:23.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 76 16 24146 24130 1.24014 1.10938 0.0252866 0.0503697 2026-02-26T03:33:24.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 77 16 24432 24416 1.23854 1.11719 0.0383912 0.0504336 2026-02-26T03:33:25.046 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 78 16 24738 24722 1.23798 1.19531 0.0414028 0.0504611 2026-02-26T03:33:26.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 79 16 25013 24997 1.23591 1.07422 0.0721448 0.050543 2026-02-26T03:33:27.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:33:27.047144+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0505835 2026-02-26T03:33:27.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:33:27.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 80 16 25310 25294 1.23496 1.16016 0.0565688 0.0505835 2026-02-26T03:33:28.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 81 16 25599 25583 1.23365 1.12891 0.0334043 0.0506405 2026-02-26T03:33:29.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 82 16 25880 25864 1.23199 1.09766 0.0396567 0.0506865 2026-02-26T03:33:30.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 83 16 26162 26146 1.23042 1.10156 0.0657399 0.0507641 2026-02-26T03:33:31.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 84 16 26454 26438 1.22935 1.14062 0.0734062 0.0508126 2026-02-26T03:33:32.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 85 16 26726 26710 1.22738 1.0625 0.0480651 0.0508953 2026-02-26T03:33:33.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 86 16 27013 26997 1.22615 1.12109 0.0815612 0.0509495 2026-02-26T03:33:34.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 87 16 27302 27286 1.22503 1.12891 0.0773877 0.0509891 2026-02-26T03:33:35.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 88 16 27588 27572 1.2238 1.11719 0.0424547 0.0510435 2026-02-26T03:33:36.047 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 89 16 27874 27858 1.2226 1.11719 0.0467577 0.0510988 2026-02-26T03:33:37.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 90 16 28148 28132 1.22091 1.07031 0.0670736 0.0511674 2026-02-26T03:33:38.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 91 16 28449 28433 1.22041 1.17578 0.0765768 0.0511946 2026-02-26T03:33:39.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 92 16 28709 28693 1.21819 1.01562 0.0605309 0.0512835 2026-02-26T03:33:40.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 93 16 28992 28976 1.21697 1.10547 0.0322621 0.0513315 2026-02-26T03:33:41.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 94 16 29273 29257 1.2157 1.09766 0.0474226 0.0513907 2026-02-26T03:33:42.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 95 16 29562 29546 1.21479 1.12891 0.0861711 0.0514297 2026-02-26T03:33:43.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 96 16 29841 29825 1.21349 1.08984 0.0425589 0.0514867 2026-02-26T03:33:44.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 97 16 30123 30107 1.21233 1.10156 0.086515 0.0515323 2026-02-26T03:33:45.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 98 16 30403 30387 1.21112 1.09375 0.0437582 0.0515846 2026-02-26T03:33:46.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 99 16 30690 30674 1.21021 1.12109 0.0441647 0.0516247 2026-02-26T03:33:47.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:33:47.048731+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0516767 2026-02-26T03:33:47.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:33:47.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 100 16 30963 30947 1.20877 1.06641 0.0374632 0.0516767 2026-02-26T03:33:48.048 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 101 16 31245 31229 1.20771 1.10156 0.0429254 0.0517298 2026-02-26T03:33:49.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 102 16 31511 31495 1.20605 1.03906 0.0357899 0.0517977 2026-02-26T03:33:50.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 103 16 31786 31770 1.20477 1.07422 0.0480901 0.0518615 2026-02-26T03:33:51.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 104 16 32055 32039 1.20329 1.05078 0.0328328 0.0519197 2026-02-26T03:33:52.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 105 16 32326 32310 1.20191 1.05859 0.0461581 0.0519792 2026-02-26T03:33:53.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 106 16 32594 32578 1.20045 1.04688 0.0336014 0.0520442 2026-02-26T03:33:54.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 107 16 32871 32855 1.19934 1.08203 0.0466108 0.0520903 2026-02-26T03:33:55.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 108 16 33137 33121 1.19786 1.03906 0.0544607 0.0521583 2026-02-26T03:33:56.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 109 16 33408 33392 1.19658 1.05859 0.0832259 0.0522137 2026-02-26T03:33:57.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 110 16 33659 33643 1.19461 0.980469 0.0724727 0.0522987 2026-02-26T03:33:58.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 111 16 33931 33915 1.19342 1.0625 0.133559 0.0523441 2026-02-26T03:33:59.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 112 16 34191 34175 1.19183 1.01562 0.0889105 0.0524199 2026-02-26T03:34:00.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 113 16 34480 34464 1.19128 1.12891 0.0810274 0.0524477 2026-02-26T03:34:01.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 114 16 34732 34716 1.18946 0.984375 0.0406273 0.052524 2026-02-26T03:34:02.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 115 16 34975 34959 1.18737 0.949219 0.0556676 0.0526079 2026-02-26T03:34:03.049 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 116 16 35247 35231 1.18629 1.0625 0.0252515 0.0526662 2026-02-26T03:34:04.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 117 16 35503 35487 1.1847 1 0.094059 0.0527374 2026-02-26T03:34:05.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 118 16 35783 35767 1.18393 1.09375 0.0274604 0.0527723 2026-02-26T03:34:06.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 119 16 36033 36017 1.18219 0.976562 0.0553541 0.0528458 2026-02-26T03:34:07.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:34:07.050328+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0529066 2026-02-26T03:34:07.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:34:07.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 120 16 36298 36282 1.18096 1.03516 0.0766678 0.0529066 2026-02-26T03:34:08.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 121 16 36564 36548 1.17979 1.03906 0.0696685 0.0529535 2026-02-26T03:34:09.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 122 16 36828 36812 1.17857 1.03125 0.0406261 0.0530136 2026-02-26T03:34:10.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 123 16 37096 37080 1.1775 1.04688 0.0837683 0.0530587 2026-02-26T03:34:11.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 124 16 37357 37341 1.17622 1.01953 0.0571149 0.0531201 2026-02-26T03:34:12.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 125 16 37606 37590 1.17459 0.972656 0.031519 0.0531892 2026-02-26T03:34:13.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 126 16 37876 37860 1.17364 1.05469 0.0281818 0.0532319 2026-02-26T03:34:14.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 127 16 38138 38122 1.17246 1.02344 0.0685965 0.0532905 2026-02-26T03:34:15.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 128 16 38397 38381 1.1712 1.01172 0.0893089 0.0533409 2026-02-26T03:34:16.050 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 129 16 38653 38637 1.16987 1 0.0522899 0.0534121 2026-02-26T03:34:17.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 130 16 38882 38866 1.16776 0.894531 0.0793839 0.0535017 2026-02-26T03:34:18.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 131 16 39133 39117 1.16633 0.980469 0.0383093 0.0535689 2026-02-26T03:34:19.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 132 16 39394 39378 1.16521 1.01953 0.0416998 0.0536216 2026-02-26T03:34:20.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 133 16 39642 39626 1.16374 0.96875 0.0722741 0.053684 2026-02-26T03:34:21.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 134 16 39905 39889 1.16272 1.02734 0.0305905 0.0537324 2026-02-26T03:34:22.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 135 16 40172 40156 1.16183 1.04297 0.061196 0.0537757 2026-02-26T03:34:23.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 136 16 40413 40397 1.16021 0.941406 0.040501 0.0538541 2026-02-26T03:34:24.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 137 16 40679 40663 1.15932 1.03906 0.0549539 0.0538963 2026-02-26T03:34:25.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 138 16 40923 40907 1.15783 0.953125 0.0691956 0.0539566 2026-02-26T03:34:26.051 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 139 16 41174 41158 1.15655 0.980469 0.035664 0.0540241 2026-02-26T03:34:27.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:34:27.051852+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0540686 2026-02-26T03:34:27.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:34:27.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 140 16 41435 41419 1.15557 1.01953 0.0421611 0.0540686 2026-02-26T03:34:28.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 141 16 41697 41681 1.15463 1.02344 0.0417079 0.0541147 2026-02-26T03:34:29.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 142 16 41940 41924 1.15319 0.949219 0.0353031 0.0541786 2026-02-26T03:34:30.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 143 16 42195 42179 1.15209 0.996094 0.0413183 0.0542323 2026-02-26T03:34:31.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 144 16 42449 42433 1.15098 0.992188 0.0542185 0.0542807 2026-02-26T03:34:32.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 145 16 42697 42681 1.14972 0.96875 0.114495 0.0543429 2026-02-26T03:34:33.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 146 16 42967 42951 1.14907 1.05469 0.0493878 0.0543803 2026-02-26T03:34:34.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 147 16 43218 43202 1.14792 0.980469 0.0614589 0.0544284 2026-02-26T03:34:35.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 148 16 43468 43452 1.14676 0.976562 0.0401664 0.054481 2026-02-26T03:34:36.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 149 16 43717 43701 1.14559 0.972656 0.071587 0.0545355 2026-02-26T03:34:37.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 150 16 43975 43959 1.14468 1.00781 0.0303741 0.054581 2026-02-26T03:34:38.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 151 16 44216 44200 1.14333 0.941406 0.0306247 0.0546483 2026-02-26T03:34:39.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 152 16 44478 44462 1.14254 1.02344 0.047049 0.0546882 2026-02-26T03:34:40.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 153 16 44723 44707 1.14133 0.957031 0.094293 0.0547398 2026-02-26T03:34:41.052 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 154 16 44958 44942 1.13988 0.917969 0.0534557 0.0548116 2026-02-26T03:34:42.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 155 16 45209 45193 1.13885 0.980469 0.0606736 0.0548671 2026-02-26T03:34:43.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 156 16 45458 45442 1.13778 0.972656 0.0544326 0.0549148 2026-02-26T03:34:44.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 157 16 45707 45691 1.13673 0.972656 0.0342514 0.0549704 2026-02-26T03:34:45.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 158 16 45950 45934 1.13554 0.949219 0.0372577 0.0550191 2026-02-26T03:34:46.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 159 16 46221 46205 1.13506 1.05859 0.0357267 0.0550482 2026-02-26T03:34:47.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:34:47.053463+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0550882 2026-02-26T03:34:47.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:34:47.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 160 16 46481 46465 1.13431 1.01562 0.0427568 0.0550882 2026-02-26T03:34:48.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 161 16 46727 46711 1.13323 0.960938 0.0611777 0.0551285 2026-02-26T03:34:49.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 162 16 46973 46957 1.13217 0.960938 0.0678894 0.0551869 2026-02-26T03:34:50.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 163 16 47207 47191 1.13083 0.914062 0.082783 0.0552508 2026-02-26T03:34:51.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 164 16 47460 47444 1.12996 0.988281 0.0895882 0.0552996 2026-02-26T03:34:52.053 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 165 16 47700 47684 1.12879 0.9375 0.0937853 0.0553469 2026-02-26T03:34:53.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 166 16 47948 47932 1.12783 0.96875 0.0723559 0.0554018 2026-02-26T03:34:54.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 167 16 48195 48179 1.12685 0.964844 0.0504288 0.0554477 2026-02-26T03:34:55.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 168 16 48442 48426 1.12589 0.964844 0.0958688 0.0554931 2026-02-26T03:34:56.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 169 16 48680 48664 1.12473 0.929688 0.098402 0.0555456 2026-02-26T03:34:57.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 170 16 48921 48905 1.12365 0.941406 0.0512551 0.055608 2026-02-26T03:34:58.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 171 16 49152 49136 1.12235 0.902344 0.0561268 0.0556715 2026-02-26T03:34:59.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 172 16 49394 49378 1.12132 0.945312 0.108486 0.0557197 2026-02-26T03:35:00.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 173 16 49648 49632 1.12058 0.992188 0.0306574 0.055762 2026-02-26T03:35:01.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 174 16 49883 49867 1.11941 0.917969 0.0417745 0.0558175 2026-02-26T03:35:02.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 175 16 50128 50112 1.11848 0.957031 0.047563 0.0558669 2026-02-26T03:35:03.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 176 16 50359 50343 1.11725 0.902344 0.0897088 0.0559163 2026-02-26T03:35:04.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 177 16 50586 50570 1.11595 0.886719 0.0423069 0.0559906 2026-02-26T03:35:05.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 178 16 50820 50804 1.11482 0.914062 0.0883604 0.0560453 2026-02-26T03:35:06.054 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 179 16 51067 51051 1.11398 0.964844 0.056173 0.0560906 2026-02-26T03:35:07.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:35:07.054964+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0561471 2026-02-26T03:35:07.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:35:07.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 180 16 51305 51289 1.11295 0.929688 0.0590229 0.0561471 2026-02-26T03:35:08.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 181 16 51539 51523 1.11186 0.914062 0.0670172 0.0561991 2026-02-26T03:35:09.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 182 16 51778 51762 1.11088 0.933594 0.0509183 0.056247 2026-02-26T03:35:10.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 183 16 52019 52003 1.10995 0.941406 0.0965165 0.0562918 2026-02-26T03:35:11.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 184 16 52262 52246 1.10908 0.949219 0.0339326 0.0563398 2026-02-26T03:35:12.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 185 16 52480 52464 1.10768 0.851562 0.0717049 0.0564046 2026-02-26T03:35:13.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 186 16 52721 52705 1.10679 0.941406 0.074623 0.0564552 2026-02-26T03:35:14.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 187 16 52961 52945 1.10588 0.9375 0.0490079 0.056502 2026-02-26T03:35:15.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 188 16 53199 53183 1.10495 0.929688 0.0507997 0.0565472 2026-02-26T03:35:16.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 189 16 53425 53409 1.10377 0.882812 0.0513774 0.0566025 2026-02-26T03:35:17.055 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 190 16 53658 53642 1.10275 0.910156 0.045489 0.0566634 2026-02-26T03:35:18.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 191 16 53889 53873 1.1017 0.902344 0.0380971 0.0567048 2026-02-26T03:35:19.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 192 16 54120 54104 1.10066 0.902344 0.102787 0.0567637 2026-02-26T03:35:20.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 193 16 54345 54329 1.09951 0.878906 0.0445125 0.0568295 2026-02-26T03:35:21.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 194 16 54576 54560 1.0985 0.902344 0.0505644 0.0568803 2026-02-26T03:35:22.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 195 16 54808 54792 1.09751 0.90625 0.0538165 0.0569328 2026-02-26T03:35:23.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 196 16 55040 55024 1.09653 0.90625 0.0764851 0.0569841 2026-02-26T03:35:24.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 197 16 55270 55254 1.09553 0.898438 0.0737908 0.0570364 2026-02-26T03:35:25.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 198 16 55486 55470 1.09426 0.84375 0.101791 0.0570991 2026-02-26T03:35:26.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 199 16 55722 55706 1.09339 0.921875 0.0510432 0.0571447 2026-02-26T03:35:27.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:35:27.056541+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0572097 2026-02-26T03:35:27.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:35:27.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 200 16 55939 55923 1.09216 0.847656 0.0549364 0.0572097 2026-02-26T03:35:28.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 201 16 56158 56142 1.09098 0.855469 0.0728797 0.0572748 2026-02-26T03:35:29.056 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 202 16 56382 56366 1.08991 0.875 0.0590861 0.0573305 2026-02-26T03:35:30.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 203 16 56612 56596 1.08897 0.898438 0.0449494 0.0573791 2026-02-26T03:35:31.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 204 16 56836 56820 1.08792 0.875 0.030054 0.0574268 2026-02-26T03:35:32.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 205 16 57062 57046 1.08692 0.882812 0.0465598 0.0574868 2026-02-26T03:35:33.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 206 16 57287 57271 1.08591 0.878906 0.0933258 0.0575392 2026-02-26T03:35:34.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 207 16 57514 57498 1.08495 0.886719 0.0409839 0.0575933 2026-02-26T03:35:35.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 208 16 57731 57715 1.08381 0.847656 0.0282036 0.0576502 2026-02-26T03:35:36.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 209 16 57967 57951 1.08303 0.921875 0.126065 0.057691 2026-02-26T03:35:37.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 210 16 58186 58170 1.08195 0.855469 0.0384341 0.0577514 2026-02-26T03:35:38.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 211 16 58415 58399 1.08106 0.894531 0.0483393 0.0578005 2026-02-26T03:35:39.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 212 16 58638 58622 1.08007 0.871094 0.0810508 0.0578536 2026-02-26T03:35:40.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 213 16 58867 58851 1.0792 0.894531 0.0496088 0.0579 2026-02-26T03:35:41.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 214 16 59089 59073 1.0782 0.867188 0.0562307 0.0579545 2026-02-26T03:35:42.057 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 215 16 59311 59295 1.07722 0.867188 0.0548711 0.0580053 2026-02-26T03:35:43.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 216 16 59547 59531 1.0765 0.921875 0.0528948 0.0580467 2026-02-26T03:35:44.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 217 16 59785 59769 1.07583 0.929688 0.0416582 0.058084 2026-02-26T03:35:45.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 218 16 60019 60003 1.07508 0.914062 0.0570739 0.0581245 2026-02-26T03:35:46.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 219 16 60247 60231 1.07424 0.890625 0.0802151 0.0581672 2026-02-26T03:35:47.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:35:47.058091+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0582153 2026-02-26T03:35:47.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:35:47.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 220 16 60469 60453 1.0733 0.867188 0.127835 0.0582153 2026-02-26T03:35:48.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 221 16 60701 60685 1.07254 0.90625 0.169108 0.0582625 2026-02-26T03:35:49.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 222 16 60926 60910 1.07167 0.878906 0.0707187 0.0583072 2026-02-26T03:35:50.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 223 16 61162 61146 1.071 0.921875 0.0767293 0.0583428 2026-02-26T03:35:51.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 224 16 61388 61372 1.07016 0.882812 0.0441262 0.0583892 2026-02-26T03:35:52.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 225 16 61613 61597 1.06931 0.878906 0.0990802 0.0584356 2026-02-26T03:35:53.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 226 16 61824 61808 1.06822 0.824219 0.0815772 0.0584928 2026-02-26T03:35:54.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 227 16 62054 62038 1.06748 0.898438 0.0427464 0.0585364 2026-02-26T03:35:55.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 228 16 62286 62270 1.06677 0.90625 0.0877129 0.0585763 2026-02-26T03:35:56.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 229 16 62508 62492 1.0659 0.867188 0.043345 0.0586221 2026-02-26T03:35:57.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 230 16 62723 62707 1.06491 0.839844 0.0546556 0.0586798 2026-02-26T03:35:58.058 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 231 16 62939 62923 1.06396 0.84375 0.0820581 0.0587257 2026-02-26T03:35:59.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 232 16 63171 63155 1.06328 0.90625 0.0776318 0.0587691 2026-02-26T03:36:00.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 233 16 63390 63374 1.06238 0.855469 0.100273 0.0588169 2026-02-26T03:36:01.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 234 16 63609 63593 1.0615 0.855469 0.0523817 0.0588631 2026-02-26T03:36:02.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 235 16 63827 63811 1.0606 0.851562 0.0540104 0.0589146 2026-02-26T03:36:03.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 236 16 64053 64037 1.05985 0.882812 0.0723524 0.0589588 2026-02-26T03:36:04.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 237 16 64263 64247 1.05884 0.820312 0.0461447 0.0590119 2026-02-26T03:36:05.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 238 16 64474 64458 1.05785 0.824219 0.0468612 0.0590652 2026-02-26T03:36:06.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 239 16 64686 64670 1.05689 0.828125 0.0550437 0.0591228 2026-02-26T03:36:07.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:36:07.059642+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0591809 2026-02-26T03:36:07.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:36:07.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 240 16 64889 64873 1.05579 0.792969 0.0986554 0.0591809 2026-02-26T03:36:08.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 241 16 65113 65097 1.05504 0.875 0.0701325 0.0592265 2026-02-26T03:36:09.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 242 16 65327 65311 1.05414 0.835938 0.0764641 0.0592754 2026-02-26T03:36:10.059 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 243 16 65544 65528 1.05329 0.847656 0.0413497 0.0593253 2026-02-26T03:36:11.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 244 16 65765 65749 1.05251 0.863281 0.0611843 0.0593682 2026-02-26T03:36:12.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 245 16 65965 65949 1.0514 0.78125 0.078703 0.0594296 2026-02-26T03:36:13.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 246 16 66183 66167 1.05059 0.851562 0.0766402 0.059477 2026-02-26T03:36:14.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 247 16 66394 66378 1.04967 0.824219 0.0860773 0.0595281 2026-02-26T03:36:15.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 248 16 66606 66590 1.04878 0.828125 0.120688 0.0595755 2026-02-26T03:36:16.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 249 16 66806 66790 1.0477 0.78125 0.0669312 0.0596382 2026-02-26T03:36:17.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 250 16 67019 67003 1.04684 0.832031 0.107573 0.0596922 2026-02-26T03:36:18.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 251 16 67229 67213 1.04594 0.820312 0.0686491 0.0597408 2026-02-26T03:36:19.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 252 16 67440 67424 1.04506 0.824219 0.0310604 0.0597913 2026-02-26T03:36:20.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 253 16 67655 67639 1.04425 0.839844 0.0828933 0.0598392 2026-02-26T03:36:21.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 254 16 67866 67850 1.04338 0.824219 0.130158 0.0598871 2026-02-26T03:36:22.060 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 255 16 68085 68069 1.04264 0.855469 0.0718384 0.0599273 2026-02-26T03:36:23.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 256 16 68290 68274 1.0417 0.800781 0.133972 0.0599859 2026-02-26T03:36:24.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 257 16 68501 68485 1.04085 0.824219 0.0439311 0.0600298 2026-02-26T03:36:25.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 258 16 68686 68670 1.03962 0.722656 0.103854 0.0601019 2026-02-26T03:36:26.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 259 16 68896 68880 1.03877 0.820312 0.100079 0.0601491 2026-02-26T03:36:27.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:36:27.061197+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0602036 2026-02-26T03:36:27.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:36:27.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 260 16 69105 69089 1.03791 0.816406 0.142948 0.0602036 2026-02-26T03:36:28.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 261 16 69311 69295 1.03702 0.804688 0.0583335 0.0602557 2026-02-26T03:36:29.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 262 16 69526 69510 1.03627 0.839844 0.0479721 0.0603002 2026-02-26T03:36:30.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 263 16 69742 69726 1.03554 0.84375 0.077317 0.0603406 2026-02-26T03:36:31.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 264 16 69944 69928 1.0346 0.789062 0.102695 0.0603928 2026-02-26T03:36:32.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 265 16 70151 70135 1.03375 0.808594 0.0930002 0.0604439 2026-02-26T03:36:33.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 266 16 70363 70347 1.03298 0.828125 0.0580878 0.0604924 2026-02-26T03:36:34.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 267 16 70558 70542 1.03196 0.761719 0.0407996 0.0605512 2026-02-26T03:36:35.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 268 16 70760 70744 1.03105 0.789062 0.0656889 0.0606036 2026-02-26T03:36:36.061 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 269 16 70968 70952 1.03024 0.8125 0.0818462 0.0606502 2026-02-26T03:36:37.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 270 16 71175 71159 1.02942 0.808594 0.0923523 0.0607019 2026-02-26T03:36:38.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 271 16 71379 71363 1.02856 0.796875 0.0964205 0.0607453 2026-02-26T03:36:39.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 272 16 71585 71569 1.02774 0.804688 0.0618237 0.0608038 2026-02-26T03:36:40.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 273 16 71786 71770 1.02685 0.785156 0.11884 0.0608542 2026-02-26T03:36:41.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 274 16 71996 71980 1.02609 0.820312 0.0963265 0.0609007 2026-02-26T03:36:42.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 275 16 72196 72180 1.0252 0.78125 0.106423 0.0609494 2026-02-26T03:36:43.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 276 16 72404 72388 1.02443 0.8125 0.104718 0.0609978 2026-02-26T03:36:44.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 277 16 72607 72591 1.0236 0.792969 0.0759673 0.0610488 2026-02-26T03:36:45.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 278 16 72807 72791 1.02273 0.78125 0.11301 0.0610951 2026-02-26T03:36:46.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 279 16 73007 72991 1.02186 0.78125 0.0877355 0.0611517 2026-02-26T03:36:47.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:36:47.062655+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0611933 2026-02-26T03:36:47.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:36:47.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 280 16 73218 73202 1.02115 0.824219 0.0474776 0.0611933 2026-02-26T03:36:48.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 281 16 73420 73404 1.02033 0.789062 0.0827606 0.0612362 2026-02-26T03:36:49.062 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 282 16 73607 73591 1.0193 0.730469 0.0745775 0.0612987 2026-02-26T03:36:50.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 283 16 73813 73797 1.01854 0.804688 0.10103 0.0613492 2026-02-26T03:36:51.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 284 16 74004 73988 1.01758 0.746094 0.0457497 0.0614064 2026-02-26T03:36:52.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 285 16 74212 74196 1.01686 0.8125 0.119213 0.061449 2026-02-26T03:36:53.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 286 16 74418 74402 1.01612 0.804688 0.0735052 0.0614931 2026-02-26T03:36:54.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 287 16 74628 74612 1.01544 0.820312 0.0765232 0.0615393 2026-02-26T03:36:55.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 288 16 74831 74815 1.01466 0.792969 0.0327758 0.0615831 2026-02-26T03:36:56.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 289 16 75030 75014 1.01384 0.777344 0.0763289 0.0616367 2026-02-26T03:36:57.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 290 16 75233 75217 1.01308 0.792969 0.154333 0.061677 2026-02-26T03:36:58.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 291 16 75434 75418 1.0123 0.785156 0.0550374 0.0617281 2026-02-26T03:36:59.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 292 16 75640 75624 1.01159 0.804688 0.0590554 0.0617679 2026-02-26T03:37:00.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 293 16 75845 75829 1.01087 0.800781 0.058285 0.0618151 2026-02-26T03:37:01.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 294 16 76035 76019 1.00995 0.742188 0.0780385 0.0618715 2026-02-26T03:37:02.063 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 295 16 76239 76223 1.00923 0.796875 0.0634784 0.0619147 2026-02-26T03:37:03.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 296 16 76434 76418 1.00839 0.761719 0.146623 0.0619675 2026-02-26T03:37:04.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 297 16 76636 76620 1.00766 0.789062 0.0303899 0.0620116 2026-02-26T03:37:05.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 298 16 76833 76817 1.00686 0.769531 0.0939547 0.0620611 2026-02-26T03:37:06.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 299 16 77037 77021 1.00615 0.796875 0.0863956 0.062106 2026-02-26T03:37:07.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout:2026-02-26T03:37:07.064176+0000 min lat: 0.0103859 max lat: 0.423703 avg lat: 0.0621437 2026-02-26T03:37:07.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-26T03:37:07.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout: 300 11 77244 77233 1.00556 0.828125 0.0707037 0.0621437 2026-02-26T03:37:08.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Total time run: 300.059 2026-02-26T03:37:08.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Total writes made: 77244 2026-02-26T03:37:08.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Write size: 4096 2026-02-26T03:37:08.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Object size: 4096 2026-02-26T03:37:08.064 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Bandwidth (MB/sec): 1.00558 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Stddev Bandwidth: 0.171387 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Max bandwidth (MB/sec): 1.40234 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Min bandwidth (MB/sec): 0.722656 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Average IOPS: 257 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Stddev IOPS: 43.875 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Max IOPS: 359 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Min IOPS: 185 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Average Latency(s): 0.0621448 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Stddev Latency(s): 0.0304544 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Max latency(s): 0.423703 2026-02-26T03:37:08.065 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Min latency(s): 0.0103859 2026-02-26T03:37:08.091 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Cleaning up (deleting benchmark objects) 2026-02-26T03:44:54.090 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Removed 77244 objects 2026-02-26T03:44:54.090 INFO:tasks.radosbench.radosbench.0.trial106.stdout:Clean up completed and total clean up time :465.999 2026-02-26T03:44:55.752 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2026-02-26T03:44:55.753 DEBUG:teuthology.orchestra.run.trial106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2026-02-26T03:44:55.838 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:44:55.833+0000 7f9a80b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:55.839 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:44:55.839+0000 7f9a80b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:56.932 INFO:teuthology.orchestra.run.trial106.stderr:pool 'unique_pool_0' removed 2026-02-26T03:44:56.943 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-26T03:44:56.958 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-26T03:44:56.958 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-26T03:44:56.958 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial106.front.sepia.ceph.com 2026-02-26T03:44:56.958 DEBUG:teuthology.orchestra.run.trial106:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:44:56.977 DEBUG:teuthology.orchestra.run.trial106:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:44:57.033 DEBUG:teuthology.orchestra.run.trial106:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:44:57.090 INFO:teuthology.orchestra.run.trial106.stdout:"600" 2026-02-26T03:44:57.090 DEBUG:teuthology.orchestra.run.trial106:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:32:04:308144 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:44:57.147 DEBUG:teuthology.orchestra.run.trial106:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:44:57.212 DEBUG:teuthology.orchestra.run.trial106:> rm /home/ubuntu/.ssh/config 2026-02-26T03:44:57.277 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:44:57.288 DEBUG:teuthology.orchestra.run.trial106:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:44:57.410 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:44:57.406+0000 7f57e3b64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:57.412 INFO:teuthology.orchestra.run.trial106.stderr:2026-02-26T03:44:57.412+0000 7f57e3b64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:44:57.559 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:44:57.559 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:44:57.559 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:44:57.559 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:44:57.654 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:44:57.654 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:44:57.654 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:44:57.746 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:44:57.747 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:44:57.747 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:44:57.817 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:44:57.817 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:44:57.817 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:44:57.817 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:44:57.853 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:44:57.853 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:44:57.853 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:44:57.853 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:44:57.902 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:44:57.902 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:44:57.902 DEBUG:teuthology.orchestra.run.trial106:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-26T03:44:57.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2026-02-26T03:44:57.970 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:44:57.970 DEBUG:teuthology.orchestra.run.trial106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:44:58.115 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:44:58.115 DEBUG:teuthology.orchestra.run.trial106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:44:58.149 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:44:58.149 DEBUG:teuthology.orchestra.run.trial106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:44:58.218 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:44:58.220 DEBUG:teuthology.misc:Transferring archived files from trial106:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71180/data/mon.a.tgz 2026-02-26T03:44:58.220 DEBUG:teuthology.orchestra.run.trial106:> mktemp 2026-02-26T03:44:58.276 INFO:teuthology.orchestra.run.trial106.stdout:/tmp/tmp.OmRbBZGdIX 2026-02-26T03:44:58.276 DEBUG:teuthology.orchestra.run.trial106:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.OmRbBZGdIX 2026-02-26T03:44:58.367 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod 0666 /tmp/tmp.OmRbBZGdIX 2026-02-26T03:44:58.450 DEBUG:teuthology.orchestra.remote:trial106:/tmp/tmp.OmRbBZGdIX is 201KB 2026-02-26T03:44:58.502 DEBUG:teuthology.orchestra.run.trial106:> rm -fr /tmp/tmp.OmRbBZGdIX 2026-02-26T03:44:58.517 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:44:58.518 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:44:58.637 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:44:58.637 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:44:58.640 DEBUG:teuthology.misc:Transferring archived files from trial106:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71180/remote/trial106/crash 2026-02-26T03:44:58.642 DEBUG:teuthology.orchestra.run.trial106:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:44:58.701 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:44:58.701 DEBUG:teuthology.orchestra.run.trial106:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:44:58.724 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:44:58.725 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:44:58.725 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2026-02-26T03:44:58.725 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:44:58.725 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2026-02-26T03:44:58.725 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2026-02-26T03:44:58.726 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41933.log 2026-02-26T03:44:58.726 INFO:teuthology.orchestra.run.trial106.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T03:44:58.726 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41662.log 2026-02-26T03:44:58.726 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2026-02-26T03:44:58.726 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2026-02-26T03:44:58.726 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41933.log.gz 2026-02-26T03:44:58.726 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41335.log 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41662.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41662.log.gz 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41711.log 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41335.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41335.log.gz 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42103.log 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41613.log 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41711.log.gz 2026-02-26T03:44:58.727 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42152.log 2026-02-26T03:44:58.728 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.42103.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42103.log.gz 2026-02-26T03:44:58.728 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41812.log 2026-02-26T03:44:58.728 INFO:teuthology.orchestra.run.trial106.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2026-02-26T03:44:58.728 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.42152.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42152.log.gz 2026-02-26T03:44:58.728 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39273.log 2026-02-26T03:44:58.728 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42808.log 2026-02-26T03:44:58.728 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41812.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41812.log.gz 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph.tmp-client.admin.39273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41934.log 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39273.log.gz 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.42808.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.42363.log -- replaced with /var/log/ceph/ceph-client.admin.42808.log.gz 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42438.log 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41934.log.gz 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.42363.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:44:58.729 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42363.log.gz 2026-02-26T03:44:58.730 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.0.42438.log: gzip 65.9% -5 -- replaced with /var/log/ceph/ceph-client.0.42438.log.gz --verbose 2026-02-26T03:44:58.730 INFO:teuthology.orchestra.run.trial106.stderr: -- /var/log/ceph/ceph-client.admin.41188.log 2026-02-26T03:44:58.730 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42054.log 2026-02-26T03:44:58.730 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40845.log 2026-02-26T03:44:58.730 INFO:teuthology.orchestra.run.trial106.stderr: 86.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:44:58.730 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41188.log.gz 2026-02-26T03:44:58.730 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.42054.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42054.log.gz 2026-02-26T03:44:58.731 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:44:58.731 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.40845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41158.log 2026-02-26T03:44:58.731 INFO:teuthology.orchestra.run.trial106.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40845.log.gz 2026-02-26T03:44:58.731 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2026-02-26T03:44:58.731 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41158.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.41158.log.gz 2026-02-26T03:44:58.731 INFO:teuthology.orchestra.run.trial106.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40566.log 2026-02-26T03:44:58.731 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42623.log 2026-02-26T03:44:58.732 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41499.log: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2026-02-26T03:44:58.732 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.40566.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40566.log.gz 2026-02-26T03:44:58.732 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40892.log 2026-02-26T03:44:58.732 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.42623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42415.log 2026-02-26T03:44:58.732 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42623.log.gz 2026-02-26T03:44:58.732 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.40892.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40892.log.gz 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.42415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42415.log.gz 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41564.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41474.log 2026-02-26T03:44:58.733 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40708.log 2026-02-26T03:44:58.734 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.41474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40661.log 2026-02-26T03:44:58.734 INFO:teuthology.orchestra.run.trial106.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41474.log.gz 2026-02-26T03:44:58.734 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.40708.log: gzip 49.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40708.log.gz -- 2026-02-26T03:44:58.734 INFO:teuthology.orchestra.run.trial106.stderr: /var/log/ceph/ceph.log 2026-02-26T03:44:58.734 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph-client.admin.40661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41404.log 2026-02-26T03:44:58.734 INFO:teuthology.orchestra.run.trial106.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40661.log.gz 2026-02-26T03:44:58.735 INFO:teuthology.orchestra.run.trial106.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.41404.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41404.log.gz 2026-02-26T03:44:58.735 INFO:teuthology.orchestra.run.trial106.stderr: 88.6% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:44:58.753 INFO:teuthology.orchestra.run.trial106.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:45:02.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2026-02-26T03:45:38.921 INFO:teuthology.orchestra.run.trial106.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:45:40.191 INFO:teuthology.orchestra.run.trial106.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:45:47.604 INFO:teuthology.orchestra.run.trial106.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:45:47.606 INFO:teuthology.orchestra.run.trial106.stderr: 2026-02-26T03:45:47.606 INFO:teuthology.orchestra.run.trial106.stderr:real 0m48.889s 2026-02-26T03:45:47.606 INFO:teuthology.orchestra.run.trial106.stderr:user 2m4.711s 2026-02-26T03:45:47.606 INFO:teuthology.orchestra.run.trial106.stderr:sys 0m5.606s 2026-02-26T03:45:47.607 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:45:47.607 DEBUG:teuthology.misc:Transferring archived files from trial106:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71180/remote/trial106/log 2026-02-26T03:45:47.608 DEBUG:teuthology.orchestra.run.trial106:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:45:57.183 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:45:57.251 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:45:57.263 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:45:57.263 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:45:57.289 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:45:57.301 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:45:57.301 DEBUG:teuthology.orchestra.run.trial106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:45:57.344 INFO:teuthology.orchestra.run.trial106.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:45:57.382 INFO:teuthology.orchestra.run.trial106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:45:57.382 INFO:teuthology.orchestra.run.trial106.stdout:=============================================================================== 2026-02-26T03:45:57.382 INFO:teuthology.orchestra.run.trial106.stdout:^- static.36.62.78.5.client> 4 8 377 47 +2279us[+2279us] +/- 59ms 2026-02-26T03:45:57.382 INFO:teuthology.orchestra.run.trial106.stdout:^- t1.time.bf1.yahoo.com 2 6 377 49 -33ms[ -33ms] +/- 42ms 2026-02-26T03:45:57.382 INFO:teuthology.orchestra.run.trial106.stdout:^- 198-12-95-197-host.coloc> 3 6 377 46 +398us[ +398us] +/- 41ms 2026-02-26T03:45:57.383 INFO:teuthology.orchestra.run.trial106.stdout:^* 99-28-14-242.lightspeed.> 1 8 377 117 -3414us[-3986us] +/- 14ms 2026-02-26T03:45:57.384 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:45:57.395 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:45:57.396 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:45:57.411 DEBUG:teuthology.orchestra.run.trial106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:45:57.477 DEBUG:teuthology.orchestra.run.trial106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:45:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:45:57.504 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:45:57.521 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:45:57.538 INFO:teuthology.task.internal:Duration was 2162.251503 seconds 2026-02-26T03:45:57.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:45:57.555 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:45:57.555 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:45:57.592 INFO:teuthology.orchestra.run.trial106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:45:57.973 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:45:57.973 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial106.front.sepia.ceph.com 2026-02-26T03:45:57.974 DEBUG:teuthology.orchestra.run.trial106:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:45:58.031 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:45:58.031 DEBUG:teuthology.orchestra.run.trial106:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:45:58.134 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:45:58.134 DEBUG:teuthology.orchestra.run.trial106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:45:58.158 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:45:58.159 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:45:58.159 INFO:teuthology.orchestra.run.trial106.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:45:58.159 INFO:teuthology.orchestra.run.trial106.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:45:58.160 INFO:teuthology.orchestra.run.trial106.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:45:58.169 INFO:teuthology.orchestra.run.trial106.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:45:58.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:45:58.190 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:45:58.190 DEBUG:teuthology.orchestra.run.trial106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:45:58.235 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:45:58.249 DEBUG:teuthology.orchestra.run.trial106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:45:58.302 INFO:teuthology.orchestra.run.trial106.stdout:kernel.core_pattern = core 2026-02-26T03:45:58.320 DEBUG:teuthology.orchestra.run.trial106:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:45:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:45:58.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:45:58.396 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:45:58.397 DEBUG:teuthology.misc:Transferring archived files from trial106:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71180/remote/trial106 2026-02-26T03:45:58.397 DEBUG:teuthology.orchestra.run.trial106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:45:58.453 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:45:58.453 DEBUG:teuthology.orchestra.run.trial106:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:45:58.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:45:58.520 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:45:58.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:45:58.538 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:45:58.538 DEBUG:teuthology.orchestra.run.trial106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:45:58.556 INFO:teuthology.orchestra.run.trial106.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:45 /home/ubuntu/cephtest 2026-02-26T03:45:58.558 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:45:58.576 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:45:58.617 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_omap_write} duration: 2162.25150346756 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-26T03:45:58.617 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:45:58.729 INFO:teuthology.run:pass