2026-02-24T15:33:59.103 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T15:33:59.118 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:33:59.198 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68172 branch: wip-liucm-shard-redesign-crimson-only description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/fio_4K_rand_rw} email: null first_in_suite: false flavor: debug job_id: '68172' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-liucm-shard-redesign-crimson-only ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 workunit: branch: wip-liucm-shard-redesign-crimson-only env: CRIMSON_COMPAT: '1' sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 owner: scheduled_liucm@soko04.front.sepia.ceph.com priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9834 sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-liucm-shard-redesign-crimson-only suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 targets: trial020.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmtDBaCt/yUDSVRCK9hDyJr6JZAW03lO8IPiV/PvE+qhkJc+Y4KwSfiZRdoiYMbCeTjfxm7jZcyCdNlZsBdXKY= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-24_15:28:57 tube: trial user: liucm verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-24T15:33:59.198 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa; will attempt to use it 2026-02-24T15:33:59.199 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa/tasks 2026-02-24T15:33:59.199 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-24T15:33:59.201 INFO:teuthology.task.internal:Checking packages... 2026-02-24T15:33:59.201 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1' 2026-02-24T15:33:59.201 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-24T15:33:59.201 INFO:teuthology.packaging:ref: None 2026-02-24T15:33:59.201 INFO:teuthology.packaging:tag: None 2026-02-24T15:33:59.201 INFO:teuthology.packaging:branch: wip-liucm-shard-redesign-crimson-only 2026-02-24T15:33:59.201 INFO:teuthology.packaging:sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:33:59.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-liucm-shard-redesign-crimson-only 2026-02-24T15:33:59.343 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5343.gd26e5c22 2026-02-24T15:33:59.345 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-24T15:33:59.357 INFO:teuthology.task.internal:no buildpackages task found 2026-02-24T15:33:59.357 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-24T15:33:59.370 INFO:teuthology.task.internal:Saving configuration 2026-02-24T15:33:59.378 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-24T15:33:59.389 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-24T15:33:59.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial020.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68172', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 15:30:35.707414', 'locked_by': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmtDBaCt/yUDSVRCK9hDyJr6JZAW03lO8IPiV/PvE+qhkJc+Y4KwSfiZRdoiYMbCeTjfxm7jZcyCdNlZsBdXKY='} 2026-02-24T15:33:59.448 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-24T15:33:59.459 INFO:teuthology.task.internal:roles: ubuntu@trial020.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-24T15:33:59.459 INFO:teuthology.run_tasks:Running task console_log... 2026-02-24T15:33:59.539 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7dc513499760>, signals=[15]) 2026-02-24T15:33:59.539 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-24T15:33:59.550 INFO:teuthology.task.internal:Opening connections... 2026-02-24T15:33:59.550 DEBUG:teuthology.task.internal:connecting to ubuntu@trial020.front.sepia.ceph.com 2026-02-24T15:33:59.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:33:59.611 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-24T15:33:59.622 DEBUG:teuthology.orchestra.run.trial020:> uname -m 2026-02-24T15:33:59.686 INFO:teuthology.orchestra.run.trial020.stdout:x86_64 2026-02-24T15:33:59.686 DEBUG:teuthology.orchestra.run.trial020:> cat /etc/os-release 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:NAME="CentOS Stream" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:VERSION="9" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:ID="centos" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:ID_LIKE="rhel fedora" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:VERSION_ID="9" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:PLATFORM_ID="platform:el9" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:ANSI_COLOR="0;31" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:LOGO="fedora-logo-icon" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:HOME_URL="https://centos.org/" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T15:33:59.742 INFO:teuthology.orchestra.run.trial020.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T15:33:59.743 INFO:teuthology.lock.ops:Updating trial020.front.sepia.ceph.com on lock server 2026-02-24T15:33:59.828 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-24T15:33:59.841 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-24T15:33:59.855 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-24T15:33:59.855 DEBUG:teuthology.orchestra.run.trial020:> test '!' -e /home/ubuntu/cephtest 2026-02-24T15:33:59.870 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-24T15:33:59.881 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-24T15:33:59.881 DEBUG:teuthology.orchestra.run.trial020:> test -z $(ls -A /var/lib/ceph) 2026-02-24T15:33:59.927 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-24T15:34:00.030 INFO:teuthology.run_tasks:Running task kernel... 2026-02-24T15:34:00.052 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-24T15:34:00.052 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-24T15:34:00.052 DEBUG:teuthology.orchestra.run.trial020:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T15:34:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:34:00.067 DEBUG:teuthology.orchestra.run.trial020:> uname -r 2026-02-24T15:34:00.122 INFO:teuthology.orchestra.run.trial020.stdout:5.14.0-681.el9.x86_64 2026-02-24T15:34:00.122 INFO:teuthology.task.kernel:Running kernel on trial020: 5.14.0-681.el9.x86_64 2026-02-24T15:34:00.122 DEBUG:teuthology.orchestra.run.trial020:> sudo yum install -y kernel 2026-02-24T15:34:09.231 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.9 MB 00:08 2026-02-24T15:34:12.996 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 27 MB 00:03 2026-02-24T15:34:16.470 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 8.0 MB 00:01 2026-02-24T15:34:17.694 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 20 kB 00:00 2026-02-24T15:34:18.350 INFO:teuthology.orchestra.run.trial020.stdout:Extra Packages for Enterprise Linux 35 MB/s | 20 MB 00:00 2026-02-24T15:34:21.519 INFO:teuthology.orchestra.run.trial020.stdout:lab-extras 1.6 MB/s | 50 kB 00:00 2026-02-24T15:34:22.532 INFO:teuthology.orchestra.run.trial020.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T15:34:22.532 INFO:teuthology.orchestra.run.trial020.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T15:34:22.552 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:34:22.553 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:34:22.553 INFO:teuthology.orchestra.run.trial020.stdout: Package Architecture Version Repository Size 2026-02-24T15:34:22.553 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:34:22.553 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:34:22.553 INFO:teuthology.orchestra.run.trial020.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T15:34:22.553 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout:Install 4 Packages 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 89 M 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 135 M 2026-02-24T15:34:22.554 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:34:23.787 INFO:teuthology.orchestra.run.trial020.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 981 kB/s | 901 kB 00:00 2026-02-24T15:34:24.242 INFO:teuthology.orchestra.run.trial020.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 13 MB/s | 17 MB 00:01 2026-02-24T15:34:24.798 INFO:teuthology.orchestra.run.trial020.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 21 MB/s | 40 MB 00:01 2026-02-24T15:34:25.557 INFO:teuthology.orchestra.run.trial020.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 18 MB/s | 31 MB 00:01 2026-02-24T15:34:25.557 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:34:25.557 INFO:teuthology.orchestra.run.trial020.stdout:Total 30 MB/s | 89 MB 00:03 2026-02-24T15:34:25.751 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:34:25.964 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:34:25.964 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:34:26.351 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:34:26.351 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:34:27.461 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:34:27.751 INFO:teuthology.orchestra.run.trial020.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:34:27.773 INFO:teuthology.orchestra.run.trial020.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:34:30.318 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:34:30.352 INFO:teuthology.orchestra.run.trial020.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:34:33.388 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:34:33.395 INFO:teuthology.orchestra.run.trial020.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:36.508 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:53.853 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:53.874 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:54.062 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:54.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:34:54.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:34:54.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:34:54.120 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:54.120 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:34:54.120 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:34:54.120 INFO:teuthology.orchestra.run.trial020.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.120 INFO:teuthology.orchestra.run.trial020.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.120 INFO:teuthology.orchestra.run.trial020.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.120 INFO:teuthology.orchestra.run.trial020.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.121 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:34:54.121 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:34:54.183 DEBUG:teuthology.orchestra.run.trial020:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:34:54.212 INFO:teuthology.orchestra.run.trial020.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.213 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.213 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.213 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T15:34:54.213 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T15:34:54.213 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-24T15:34:54.214 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial020.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T15:34:54.214 DEBUG:teuthology.orchestra.run.trial020:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:34:54.278 INFO:teuthology.orchestra.run.trial020.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.279 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.279 DEBUG:teuthology.orchestra.run.trial020:> sudo rpm -qi grub2-tools 2026-02-24T15:34:54.348 INFO:teuthology.orchestra.run.trial020.stdout:Name : grub2-tools 2026-02-24T15:34:54.348 INFO:teuthology.orchestra.run.trial020.stdout:Epoch : 1 2026-02-24T15:34:54.348 INFO:teuthology.orchestra.run.trial020.stdout:Version : 2.06 2026-02-24T15:34:54.348 INFO:teuthology.orchestra.run.trial020.stdout:Release : 120.el9 2026-02-24T15:34:54.348 INFO:teuthology.orchestra.run.trial020.stdout:Architecture: x86_64 2026-02-24T15:34:54.348 INFO:teuthology.orchestra.run.trial020.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Group : Unspecified 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Size : 9108611 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:License : GPLv3+ 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Packager : builder@centos.org 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Vendor : CentOS 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Summary : Support tools for GRUB. 2026-02-24T15:34:54.349 INFO:teuthology.orchestra.run.trial020.stdout:Description : 2026-02-24T15:34:54.350 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:34:54.350 INFO:teuthology.orchestra.run.trial020.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T15:34:54.350 INFO:teuthology.orchestra.run.trial020.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T15:34:54.350 INFO:teuthology.orchestra.run.trial020.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T15:34:54.350 INFO:teuthology.orchestra.run.trial020.stdout:hardware devices. 2026-02-24T15:34:54.350 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:34:54.350 INFO:teuthology.orchestra.run.trial020.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T15:34:54.351 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T15:34:54.351 INFO:teuthology.task.kernel:Updating grub on trial020 to boot 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.351 DEBUG:teuthology.orchestra.run.trial020:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T15:34:54.414 INFO:teuthology.orchestra.run.trial020.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T15:34:54.415 DEBUG:teuthology.orchestra.run.trial020:> sudo command -v grubby && echo yes || echo no 2026-02-24T15:34:54.480 INFO:teuthology.orchestra.run.trial020.stdout:/sbin/grubby 2026-02-24T15:34:54.482 INFO:teuthology.orchestra.run.trial020.stdout:yes 2026-02-24T15:34:54.482 DEBUG:teuthology.orchestra.run.trial020:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T15:34:54.545 INFO:teuthology.orchestra.run.trial020.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T15:34:54.547 DEBUG:teuthology.orchestra.run.trial020:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.618 DEBUG:teuthology.orchestra.run.trial020:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T15:34:54.823 INFO:teuthology.orchestra.run.trial020.stderr:Generating grub configuration file ... 2026-02-24T15:34:54.892 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.898 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T15:34:55.147 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T15:34:55.154 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T15:34:55.188 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T15:34:55.194 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T15:34:55.225 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T15:34:55.232 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T15:34:55.255 INFO:teuthology.orchestra.run.trial020.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T15:34:55.261 INFO:teuthology.orchestra.run.trial020.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T15:34:55.347 INFO:teuthology.orchestra.run.trial020.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T15:34:55.356 INFO:teuthology.orchestra.run.trial020.stderr:done 2026-02-24T15:34:55.358 DEBUG:teuthology.orchestra.run.trial020:> test -d /sys/firmware/efi 2026-02-24T15:34:55.374 DEBUG:teuthology.orchestra.run.trial020:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T15:34:55.439 INFO:teuthology.orchestra.run.trial020.stdout:/boot/efi/EFI/BOOT 2026-02-24T15:34:55.439 INFO:teuthology.orchestra.run.trial020.stdout:/boot/efi/EFI/centos 2026-02-24T15:34:55.441 DEBUG:teuthology.orchestra.run.trial020:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T15:34:55.504 INFO:teuthology.orchestra.run.trial020.stdout:no 2026-02-24T15:34:55.505 DEBUG:teuthology.orchestra.run.trial020:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T15:34:55.568 INFO:teuthology.orchestra.run.trial020.stdout:yes 2026-02-24T15:34:55.569 DEBUG:teuthology.orchestra.run.trial020:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T15:34:55.632 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial020 2026-02-24T15:34:55.632 DEBUG:teuthology.orchestra.run.trial020:> sudo bash -c ' 2026-02-24T15:34:55.632 DEBUG:teuthology.orchestra.run.trial020:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T15:34:55.632 DEBUG:teuthology.orchestra.run.trial020:> ' 2026-02-24T15:35:25.674 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-24T15:35:25.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T15:35:25.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:36:25.728 DEBUG:teuthology.orchestra.remote:timed out 2026-02-24T15:36:34.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T15:36:34.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:36:34.878 DEBUG:teuthology.orchestra.run.trial020:> true 2026-02-24T15:36:35.019 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T15:36:35.019 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-24T15:36:35.019 DEBUG:teuthology.orchestra.run.trial020:> uname -r 2026-02-24T15:36:35.075 INFO:teuthology.orchestra.run.trial020.stdout:5.14.0-686.el9.x86_64 2026-02-24T15:36:35.075 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T15:36:35.075 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T15:36:35.075 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T15:36:36.076 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-24T15:36:36.077 DEBUG:teuthology.orchestra.run.trial020:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T15:36:36.107 INFO:teuthology.orchestra.run.trial020.stdout:ttyS1 2026-02-24T15:36:36.138 DEBUG:teuthology.parallel:result is None 2026-02-24T15:36:36.138 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-24T15:36:36.152 INFO:teuthology.task.internal:Creating test directory... 2026-02-24T15:36:36.152 DEBUG:teuthology.orchestra.run.trial020:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T15:36:36.194 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-24T15:36:36.204 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-24T15:36:36.217 INFO:teuthology.task.internal:Creating archive directory... 2026-02-24T15:36:36.218 DEBUG:teuthology.orchestra.run.trial020:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T15:36:36.257 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-24T15:36:36.267 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-24T15:36:36.267 DEBUG:teuthology.orchestra.run.trial020:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T15:36:36.316 INFO:teuthology.orchestra.run.trial020.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:36:36.324 INFO:teuthology.orchestra.run.trial020.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:36:36.326 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-24T15:36:36.337 INFO:teuthology.task.internal:Configuring sudo... 2026-02-24T15:36:36.338 DEBUG:teuthology.orchestra.run.trial020:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T15:36:36.390 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-24T15:36:36.405 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-24T15:36:36.405 DEBUG:teuthology.orchestra.run.trial020:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T15:36:36.446 DEBUG:teuthology.orchestra.run.trial020:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:36:36.606 DEBUG:teuthology.orchestra.run.trial020:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:36:36.630 DEBUG:teuthology.orchestra.run.trial020:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:36:36.782 DEBUG:teuthology.orchestra.run.trial020:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:36:36.806 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T15:36:36.806 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T15:36:36.871 DEBUG:teuthology.orchestra.run.trial020:> sudo service rsyslog restart 2026-02-24T15:36:36.937 INFO:teuthology.orchestra.run.trial020.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T15:36:37.330 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-24T15:36:37.341 INFO:teuthology.task.internal:Starting timer... 2026-02-24T15:36:37.341 INFO:teuthology.run_tasks:Running task pcp... 2026-02-24T15:36:37.362 INFO:teuthology.run_tasks:Running task selinux... 2026-02-24T15:36:37.379 DEBUG:teuthology.orchestra.run.trial020:> sudo service auditd rotate 2026-02-24T15:36:37.408 INFO:teuthology.orchestra.run.trial020.stdout:Rotating logs: 2026-02-24T15:36:37.410 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-24T15:36:37.410 DEBUG:teuthology.orchestra.run.trial020:> /usr/sbin/getenforce 2026-02-24T15:36:37.466 INFO:teuthology.orchestra.run.trial020.stdout:Permissive 2026-02-24T15:36:37.466 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial020.front.sepia.ceph.com': 'permissive'} 2026-02-24T15:36:37.467 DEBUG:teuthology.orchestra.run.trial020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T15:36:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:36:37.530 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-24T15:36:37.530 DEBUG:teuthology.orchestra.run.trial020:> sudo /usr/sbin/setenforce permissive 2026-02-24T15:36:37.593 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-24T15:36:37.607 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-24T15:36:37.619 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-24T15:36:37.621 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial020.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-24T15:39:25.288 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial020.front.sepia.ceph.com')] 2026-02-24T15:39:25.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T15:39:25.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:39:25.395 DEBUG:teuthology.orchestra.run.trial020:> true 2026-02-24T15:39:25.472 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial020.front.sepia.ceph.com' 2026-02-24T15:39:25.472 INFO:teuthology.run_tasks:Running task clock... 2026-02-24T15:39:25.486 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-24T15:39:25.486 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T15:39:25.487 DEBUG:teuthology.orchestra.run.trial020:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T15:39:25.539 INFO:teuthology.orchestra.run.trial020.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T15:39:25.551 INFO:teuthology.orchestra.run.trial020.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T15:39:25.572 INFO:teuthology.orchestra.run.trial020.stderr:sudo: ntpd: command not found 2026-02-24T15:39:25.582 INFO:teuthology.orchestra.run.trial020.stdout:506 Cannot talk to daemon 2026-02-24T15:39:25.594 INFO:teuthology.orchestra.run.trial020.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T15:39:25.606 INFO:teuthology.orchestra.run.trial020.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T15:39:25.649 INFO:teuthology.orchestra.run.trial020.stderr:bash: line 1: ntpq: command not found 2026-02-24T15:39:25.650 INFO:teuthology.orchestra.run.trial020.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T15:39:25.651 INFO:teuthology.orchestra.run.trial020.stdout:=============================================================================== 2026-02-24T15:39:25.652 INFO:teuthology.run_tasks:Running task install... 2026-02-24T15:39:25.662 DEBUG:teuthology.task.install:project ceph 2026-02-24T15:39:25.662 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T15:39:25.662 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T15:39:25.662 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T15:39:25.666 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T15:39:25.666 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-24T15:39:25.667 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T15:39:25.668 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:39:25.807 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T15:39:25.807 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T15:39:25.940 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T15:39:25.940 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T15:39:25.940 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T15:39:25.965 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-24T15:39:25.965 DEBUG:teuthology.orchestra.run.trial020:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T15:39:26.029 DEBUG:teuthology.orchestra.run.trial020:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T15:39:26.104 DEBUG:teuthology.orchestra.run.trial020:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T15:39:26.168 INFO:teuthology.orchestra.run.trial020.stdout:check_obsoletes = 1 2026-02-24T15:39:26.169 DEBUG:teuthology.orchestra.run.trial020:> sudo yum clean all 2026-02-24T15:39:26.318 INFO:teuthology.orchestra.run.trial020.stdout:41 files removed 2026-02-24T15:39:26.332 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-radosgw 2026-02-24T15:39:26.623 INFO:teuthology.orchestra.run.trial020.stdout:ceph packages for x86_64 651 kB/s | 92 kB 00:00 2026-02-24T15:39:26.770 INFO:teuthology.orchestra.run.trial020.stdout:ceph noarch packages 200 kB/s | 24 kB 00:00 2026-02-24T15:39:26.909 INFO:teuthology.orchestra.run.trial020.stdout:ceph source packages 18 kB/s | 2.1 kB 00:00 2026-02-24T15:39:29.060 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.9 MB 00:02 2026-02-24T15:39:33.043 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 27 MB 00:03 2026-02-24T15:39:36.498 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 8.0 MB 00:01 2026-02-24T15:39:37.426 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - Extras packages 109 kB/s | 20 kB 00:00 2026-02-24T15:39:37.751 INFO:teuthology.orchestra.run.trial020.stdout:Extra Packages for Enterprise Linux 78 MB/s | 20 MB 00:00 2026-02-24T15:39:40.874 INFO:teuthology.orchestra.run.trial020.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-24T15:39:42.086 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: ceph-radosgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 11 M 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout:Upgrading: 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: librados2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 3.3 M 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: librbd1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.7 M 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: ceph-base x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.1 M 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 21 M 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: ceph-selinux x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 25 k 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: libcephfs2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 860 k 2026-02-24T15:39:42.088 INFO:teuthology.orchestra.run.trial020.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: libradosstriper1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 263 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: librgw2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.6 M 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: python3-ceph-argparse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 45 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: python3-ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 185 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: python3-cephfs x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 203 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: python3-rados x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 324 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: python3-rbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 305 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: python3-rgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 100 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T15:39:42.089 INFO:teuthology.orchestra.run.trial020.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout:Installing weak dependencies: 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout:Install 32 Packages 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout:Upgrade 2 Packages 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 60 M 2026-02-24T15:39:42.090 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:39:44.199 INFO:teuthology.orchestra.run.trial020.stdout:(1/34): ceph-base-20.3.0-5343.gd26e5c22.el9.x86 2.7 MB/s | 5.1 MB 00:01 2026-02-24T15:39:44.222 INFO:teuthology.orchestra.run.trial020.stdout:(2/34): ceph-selinux-20.3.0-5343.gd26e5c22.el9. 1.0 MB/s | 25 kB 00:00 2026-02-24T15:39:44.337 INFO:teuthology.orchestra.run.trial020.stdout:(3/34): libcephfs2-20.3.0-5343.gd26e5c22.el9.x8 7.3 MB/s | 860 kB 00:00 2026-02-24T15:39:44.367 INFO:teuthology.orchestra.run.trial020.stdout:(4/34): libradosstriper1-20.3.0-5343.gd26e5c22. 8.6 MB/s | 263 kB 00:00 2026-02-24T15:39:44.916 INFO:teuthology.orchestra.run.trial020.stdout:(5/34): librgw2-20.3.0-5343.gd26e5c22.el9.x86_6 10 MB/s | 5.6 MB 00:00 2026-02-24T15:39:44.961 INFO:teuthology.orchestra.run.trial020.stdout:(6/34): python3-ceph-argparse-20.3.0-5343.gd26e 1.0 MB/s | 45 kB 00:00 2026-02-24T15:39:45.008 INFO:teuthology.orchestra.run.trial020.stdout:(7/34): python3-ceph-common-20.3.0-5343.gd26e5c 3.9 MB/s | 185 kB 00:00 2026-02-24T15:39:45.060 INFO:teuthology.orchestra.run.trial020.stdout:(8/34): python3-cephfs-20.3.0-5343.gd26e5c22.el 3.8 MB/s | 203 kB 00:00 2026-02-24T15:39:45.118 INFO:teuthology.orchestra.run.trial020.stdout:(9/34): python3-rados-20.3.0-5343.gd26e5c22.el9 5.5 MB/s | 324 kB 00:00 2026-02-24T15:39:45.178 INFO:teuthology.orchestra.run.trial020.stdout:(10/34): python3-rbd-20.3.0-5343.gd26e5c22.el9. 4.9 MB/s | 305 kB 00:00 2026-02-24T15:39:45.237 INFO:teuthology.orchestra.run.trial020.stdout:(11/34): python3-rgw-20.3.0-5343.gd26e5c22.el9. 1.7 MB/s | 100 kB 00:00 2026-02-24T15:39:45.248 INFO:teuthology.orchestra.run.trial020.stdout:(12/34): mailcap-2.1.49-5.el9.noarch.rpm 2.9 MB/s | 33 kB 00:00 2026-02-24T15:39:45.257 INFO:teuthology.orchestra.run.trial020.stdout:(13/34): pciutils-3.7.0-7.el9.x86_64.rpm 11 MB/s | 93 kB 00:00 2026-02-24T15:39:45.322 INFO:teuthology.orchestra.run.trial020.stdout:(14/34): ceph-common-20.3.0-5343.gd26e5c22.el9. 7.0 MB/s | 21 MB 00:02 2026-02-24T15:39:45.591 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:45.648 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:45.942 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:45.970 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:46.583 INFO:teuthology.orchestra.run.trial020.stdout:(15/34): ceph-radosgw-20.3.0-5343.gd26e5c22.el9 2.6 MB/s | 11 MB 00:04 2026-02-24T15:39:46.608 INFO:teuthology.orchestra.run.trial020.stdout:(16/34): boost-program-options-1.75.0-13.el9.x8 77 kB/s | 104 kB 00:01 2026-02-24T15:39:46.689 INFO:teuthology.orchestra.run.trial020.stdout:(17/34): libnbd-1.20.3-4.el9.x86_64.rpm 120 kB/s | 164 kB 00:01 2026-02-24T15:39:46.913 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:46.958 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:47.016 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:47.236 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:47.287 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:47.349 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:47.877 INFO:teuthology.orchestra.run.trial020.stdout:(18/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 36 kB/s | 45 kB 00:01 2026-02-24T15:39:47.978 INFO:teuthology.orchestra.run.trial020.stdout:(19/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 115 kB/s | 160 kB 00:01 2026-02-24T15:39:48.237 INFO:teuthology.orchestra.run.trial020.stdout:(20/34): librdkafka-1.6.1-102.el9.x86_64.rpm 428 kB/s | 662 kB 00:01 2026-02-24T15:39:48.687 INFO:teuthology.orchestra.run.trial020.stdout:(21/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 361 kB/s | 292 kB 00:00 2026-02-24T15:39:48.742 INFO:teuthology.orchestra.run.trial020.stdout:(22/34): qatlib-25.08.0-2.el9.x86_64.rpm 314 kB/s | 240 kB 00:00 2026-02-24T15:39:48.813 INFO:teuthology.orchestra.run.trial020.stdout:(23/34): qatlib-service-25.08.0-2.el9.x86_64.rp 64 kB/s | 37 kB 00:00 2026-02-24T15:39:49.074 INFO:teuthology.orchestra.run.trial020.stdout:(24/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 928 kB/s | 308 kB 00:00 2026-02-24T15:39:49.118 INFO:teuthology.orchestra.run.trial020.stdout:(25/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 566 kB/s | 25 kB 00:00 2026-02-24T15:39:49.164 INFO:teuthology.orchestra.run.trial020.stdout:(26/34): liboath-2.6.12-1.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2026-02-24T15:39:49.213 INFO:teuthology.orchestra.run.trial020.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2026-02-24T15:39:49.299 INFO:teuthology.orchestra.run.trial020.stdout:(28/34): libarrow-9.0.0-15.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2026-02-24T15:39:49.311 INFO:teuthology.orchestra.run.trial020.stdout:(29/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 106 kB/s | 66 kB 00:00 2026-02-24T15:39:49.333 INFO:teuthology.orchestra.run.trial020.stdout:(30/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 6.8 MB/s | 838 kB 00:00 2026-02-24T15:39:49.343 INFO:teuthology.orchestra.run.trial020.stdout:(31/34): re2-20211101-20.el9.x86_64.rpm 4.2 MB/s | 191 kB 00:00 2026-02-24T15:39:49.733 INFO:teuthology.orchestra.run.trial020.stdout:(32/34): thrift-0.15.0-4.el9.x86_64.rpm 3.8 MB/s | 1.6 MB 00:00 2026-02-24T15:39:50.546 INFO:teuthology.orchestra.run.trial020.stdout:(33/34): librados2-20.3.0-5343.gd26e5c22.el9.x8 2.7 MB/s | 3.3 MB 00:01 2026-02-24T15:39:51.623 INFO:teuthology.orchestra.run.trial020.stdout:(34/34): librbd1-20.3.0-5343.gd26e5c22.el9.x86_ 1.2 MB/s | 2.7 MB 00:02 2026-02-24T15:39:51.628 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:39:51.628 INFO:teuthology.orchestra.run.trial020.stdout:Total 6.3 MB/s | 60 MB 00:09 2026-02-24T15:39:51.715 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:39:51.732 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:39:51.732 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:39:51.972 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:39:51.973 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:39:52.254 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:39:52.393 INFO:teuthology.orchestra.run.trial020.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-24T15:39:52.399 INFO:teuthology.orchestra.run.trial020.stdout: Upgrading : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:39:52.429 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:39:52.465 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:39:52.466 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:39:52.477 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:39:52.485 INFO:teuthology.orchestra.run.trial020.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-24T15:39:52.487 INFO:teuthology.orchestra.run.trial020.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-24T15:39:52.490 INFO:teuthology.orchestra.run.trial020.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-24T15:39:52.495 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-24T15:39:52.506 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 9/36 2026-02-24T15:39:52.518 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:39:52.519 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:39:52.542 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:39:52.546 INFO:teuthology.orchestra.run.trial020.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-24T15:39:52.579 INFO:teuthology.orchestra.run.trial020.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-24T15:39:52.583 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-24T15:39:52.589 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-24T15:39:52.591 INFO:teuthology.orchestra.run.trial020.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-24T15:39:52.595 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-24T15:39:52.711 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-24T15:39:52.712 INFO:teuthology.orchestra.run.trial020.stdout: Upgrading : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:39:52.738 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:39:52.741 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 20/36 2026-02-24T15:39:52.747 INFO:teuthology.orchestra.run.trial020.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-24T15:39:52.996 INFO:teuthology.orchestra.run.trial020.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-24T15:39:52.997 INFO:teuthology.orchestra.run.trial020.stdout: Installing : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:39:53.014 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:39:53.019 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 24/36 2026-02-24T15:39:53.023 INFO:teuthology.orchestra.run.trial020.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-24T15:39:53.026 INFO:teuthology.orchestra.run.trial020.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-24T15:39:53.026 INFO:teuthology.orchestra.run.trial020.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:39:53.044 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:39:53.047 INFO:teuthology.orchestra.run.trial020.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-24T15:39:53.082 INFO:teuthology.orchestra.run.trial020.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-24T15:39:53.084 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 30/36 2026-02-24T15:39:54.026 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:39:54.029 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:39:54.288 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:39:54.292 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:39:54.330 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:39:54.330 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T15:39:54.330 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T15:39:54.330 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:54.334 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout:skipping the directory /sys 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout:skipping the directory /proc 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout:skipping the directory /mnt 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout:skipping the directory /var/tmp 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout:skipping the directory /home 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout:skipping the directory /root 2026-02-24T15:39:58.795 INFO:teuthology.orchestra.run.trial020.stdout:skipping the directory /tmp 2026-02-24T15:39:58.796 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:59.288 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:39:59.311 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:39:59.311 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:39:59.311 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T15:39:59.311 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:39:59.311 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:39:59.311 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:59.313 INFO:teuthology.orchestra.run.trial020.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:39:59.325 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:39:59.326 INFO:teuthology.orchestra.run.trial020.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:39:59.380 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:39:59.380 INFO:teuthology.orchestra.run.trial020.stdout:Creating group 'qat' with GID 992. 2026-02-24T15:39:59.380 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 7/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 8/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 9/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 13/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-24T15:39:59.663 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-24T15:39:59.664 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-24T15:39:59.664 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-24T15:39:59.664 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:39:59.664 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-24T15:39:59.664 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 35/36 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout:Upgraded: 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T15:39:59.734 INFO:teuthology.orchestra.run.trial020.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:39:59.735 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:39:59.775 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-test 2026-02-24T15:40:00.093 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:20 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:40:00.176 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:40:00.176 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:40:00.176 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: ceph-test x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 126 M 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: libcephsqlite x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 153 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout:Install 12 Packages 2026-02-24T15:40:00.177 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:40:00.178 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 131 M 2026-02-24T15:40:00.178 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 981 M 2026-02-24T15:40:00.178 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:40:01.326 INFO:teuthology.orchestra.run.trial020.stdout:(1/12): libcephsqlite-20.3.0-5343.gd26e5c22.el9 167 kB/s | 153 kB 00:00 2026-02-24T15:40:01.427 INFO:teuthology.orchestra.run.trial020.stdout:(2/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.1 MB/s | 2.1 MB 00:01 2026-02-24T15:40:01.866 INFO:teuthology.orchestra.run.trial020.stdout:(3/12): libatomic-11.5.0-14.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2026-02-24T15:40:02.084 INFO:teuthology.orchestra.run.trial020.stdout:(4/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 143 kB/s | 94 kB 00:00 2026-02-24T15:40:02.453 INFO:teuthology.orchestra.run.trial020.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 894 kB/s | 525 kB 00:00 2026-02-24T15:40:02.555 INFO:teuthology.orchestra.run.trial020.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 374 kB/s | 176 kB 00:00 2026-02-24T15:40:02.555 INFO:teuthology.orchestra.run.trial020.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 2.2 MB/s | 233 kB 00:00 2026-02-24T15:40:02.741 INFO:teuthology.orchestra.run.trial020.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 5.4 MB/s | 1.0 MB 00:00 2026-02-24T15:40:02.755 INFO:teuthology.orchestra.run.trial020.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 1.5 MB/s | 303 kB 00:00 2026-02-24T15:40:02.821 INFO:teuthology.orchestra.run.trial020.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 795 kB/s | 64 kB 00:00 2026-02-24T15:40:03.235 INFO:teuthology.orchestra.run.trial020.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 253 kB/s | 122 kB 00:00 2026-02-24T15:41:32.835 INFO:teuthology.orchestra.run.trial020.stdout:(12/12): ceph-test-20.3.0-5343.gd26e5c22.el9.x8 1.4 MB/s | 126 MB 01:32 2026-02-24T15:41:32.836 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:41:32.837 INFO:teuthology.orchestra.run.trial020.stdout:Total 1.4 MB/s | 131 MB 01:32 2026-02-24T15:41:32.908 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:41:32.920 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:41:32.920 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:41:33.347 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:41:33.347 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:41:33.801 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:41:33.809 INFO:teuthology.orchestra.run.trial020.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-24T15:41:33.818 INFO:teuthology.orchestra.run.trial020.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-24T15:41:33.823 INFO:teuthology.orchestra.run.trial020.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-24T15:41:33.827 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-24T15:41:33.829 INFO:teuthology.orchestra.run.trial020.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-24T15:41:33.834 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-24T15:41:33.835 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:41:33.852 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:41:33.854 INFO:teuthology.orchestra.run.trial020.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-24T15:41:33.864 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-24T15:41:33.873 INFO:teuthology.orchestra.run.trial020.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-24T15:41:33.874 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:41:40.777 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:41:40.796 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:41:41.547 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:41:41.547 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-24T15:41:41.548 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-24T15:41:41.606 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-24T15:41:41.606 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T15:41:41.607 INFO:teuthology.orchestra.run.trial020.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T15:41:41.608 INFO:teuthology.orchestra.run.trial020.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T15:41:41.608 INFO:teuthology.orchestra.run.trial020.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T15:41:41.608 INFO:teuthology.orchestra.run.trial020.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T15:41:41.608 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:41:41.608 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:41:41.647 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph 2026-02-24T15:41:41.957 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:02:01 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:41:42.041 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:41:42.042 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:41:42.042 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:41:42.042 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:41:42.042 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:41:42.042 INFO:teuthology.orchestra.run.trial020.stdout: ceph x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 6.5 k 2026-02-24T15:41:42.042 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mds x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.0 M 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 937 k 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 297 k 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mon x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 4.9 M 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 163 k 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd-classic x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 15 M 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd-crimson x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 88 M 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T15:41:42.043 INFO:teuthology.orchestra.run.trial020.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T15:41:42.044 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-collections 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T15:41:42.045 INFO:teuthology.orchestra.run.trial020.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T15:41:42.046 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:41:42.047 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:41:42.047 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:41:42.047 INFO:teuthology.orchestra.run.trial020.stdout:Install 42 Packages 2026-02-24T15:41:42.047 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:41:42.048 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 115 M 2026-02-24T15:41:42.048 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 974 M 2026-02-24T15:41:42.048 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:41:42.585 INFO:teuthology.orchestra.run.trial020.stdout:(1/42): ceph-20.3.0-5343.gd26e5c22.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-24T15:41:44.180 INFO:teuthology.orchestra.run.trial020.stdout:(2/42): ceph-mgr-20.3.0-5343.gd26e5c22.el9.x86_ 500 kB/s | 937 kB 00:01 2026-02-24T15:41:44.458 INFO:teuthology.orchestra.run.trial020.stdout:(3/42): ceph-osd-20.3.0-5343.gd26e5c22.el9.x86_ 589 kB/s | 163 kB 00:00 2026-02-24T15:41:44.775 INFO:teuthology.orchestra.run.trial020.stdout:(4/42): ceph-mds-20.3.0-5343.gd26e5c22.el9.x86_ 839 kB/s | 2.0 MB 00:02 2026-02-24T15:41:49.968 INFO:teuthology.orchestra.run.trial020.stdout:(5/42): ceph-mon-20.3.0-5343.gd26e5c22.el9.x86_ 675 kB/s | 4.9 MB 00:07 2026-02-24T15:41:50.313 INFO:teuthology.orchestra.run.trial020.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5343.gd26e 859 kB/s | 297 kB 00:00 2026-02-24T15:41:50.596 INFO:teuthology.orchestra.run.trial020.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 143 kB/s | 40 kB 00:00 2026-02-24T15:41:50.795 INFO:teuthology.orchestra.run.trial020.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 364 kB/s | 72 kB 00:00 2026-02-24T15:41:51.033 INFO:teuthology.orchestra.run.trial020.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2026-02-24T15:41:51.324 INFO:teuthology.orchestra.run.trial020.stdout:(10/42): python3-cryptography-36.0.1-5.el9.x86_ 4.3 MB/s | 1.2 MB 00:00 2026-02-24T15:41:51.406 INFO:teuthology.orchestra.run.trial020.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2026-02-24T15:41:51.489 INFO:teuthology.orchestra.run.trial020.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 135 kB 00:00 2026-02-24T15:41:51.572 INFO:teuthology.orchestra.run.trial020.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 1.8 MB/s | 150 kB 00:00 2026-02-24T15:41:51.655 INFO:teuthology.orchestra.run.trial020.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 1.5 MB/s | 126 kB 00:00 2026-02-24T15:41:51.742 INFO:teuthology.orchestra.run.trial020.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 2.5 MB/s | 218 kB 00:00 2026-02-24T15:41:51.827 INFO:teuthology.orchestra.run.trial020.stdout:(16/42): unzip-6.0-59.el9.x86_64.rpm 2.1 MB/s | 182 kB 00:00 2026-02-24T15:41:51.913 INFO:teuthology.orchestra.run.trial020.stdout:(17/42): zip-3.0-35.el9.x86_64.rpm 3.0 MB/s | 266 kB 00:00 2026-02-24T15:41:52.678 INFO:teuthology.orchestra.run.trial020.stdout:(18/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 322 kB/s | 246 kB 00:00 2026-02-24T15:41:53.414 INFO:teuthology.orchestra.run.trial020.stdout:(19/42): lua-5.4.4-4.el9.x86_64.rpm 256 kB/s | 188 kB 00:00 2026-02-24T15:41:54.143 INFO:teuthology.orchestra.run.trial020.stdout:(20/42): python3-libstoragemgmt-1.10.1-1.el9.x8 242 kB/s | 177 kB 00:00 2026-02-24T15:41:54.781 INFO:teuthology.orchestra.run.trial020.stdout:(21/42): python3-packaging-20.9-5.el9.noarch.rp 121 kB/s | 77 kB 00:00 2026-02-24T15:41:55.363 INFO:teuthology.orchestra.run.trial020.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 72 kB/s | 42 kB 00:00 2026-02-24T15:41:55.918 INFO:teuthology.orchestra.run.trial020.stdout:(23/42): lua-devel-5.4.4-4.el9.x86_64.rpm 40 kB/s | 22 kB 00:00 2026-02-24T15:41:55.954 INFO:teuthology.orchestra.run.trial020.stdout:(24/42): luarocks-3.9.2-5.el9.noarch.rpm 4.2 MB/s | 151 kB 00:00 2026-02-24T15:41:55.957 INFO:teuthology.orchestra.run.trial020.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 7.8 MB/s | 29 kB 00:00 2026-02-24T15:41:55.961 INFO:teuthology.orchestra.run.trial020.stdout:(26/42): python3-backports-tarfile-1.2.0-1.el9. 14 MB/s | 60 kB 00:00 2026-02-24T15:41:55.966 INFO:teuthology.orchestra.run.trial020.stdout:(27/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 11 MB/s | 43 kB 00:00 2026-02-24T15:41:55.972 INFO:teuthology.orchestra.run.trial020.stdout:(28/42): python3-cheroot-10.0.1-4.el9.noarch.rp 29 MB/s | 173 kB 00:00 2026-02-24T15:41:55.979 INFO:teuthology.orchestra.run.trial020.stdout:(29/42): python3-cherrypy-18.6.1-2.el9.noarch.r 49 MB/s | 358 kB 00:00 2026-02-24T15:41:55.982 INFO:teuthology.orchestra.run.trial020.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 3.0 MB/s | 11 kB 00:00 2026-02-24T15:41:55.986 INFO:teuthology.orchestra.run.trial020.stdout:(31/42): python3-jaraco-classes-3.2.1-5.el9.noa 4.8 MB/s | 18 kB 00:00 2026-02-24T15:41:55.990 INFO:teuthology.orchestra.run.trial020.stdout:(32/42): python3-jaraco-collections-3.0.0-8.el9 6.3 MB/s | 23 kB 00:00 2026-02-24T15:41:55.994 INFO:teuthology.orchestra.run.trial020.stdout:(33/42): python3-jaraco-context-6.0.1-3.el9.noa 5.1 MB/s | 20 kB 00:00 2026-02-24T15:41:55.997 INFO:teuthology.orchestra.run.trial020.stdout:(34/42): python3-jaraco-functools-3.5.0-2.el9.n 5.5 MB/s | 19 kB 00:00 2026-02-24T15:41:56.001 INFO:teuthology.orchestra.run.trial020.stdout:(35/42): python3-jaraco-text-4.0.0-2.el9.noarch 7.2 MB/s | 26 kB 00:00 2026-02-24T15:41:56.006 INFO:teuthology.orchestra.run.trial020.stdout:(36/42): python3-more-itertools-8.12.0-2.el9.no 18 MB/s | 79 kB 00:00 2026-02-24T15:41:56.009 INFO:teuthology.orchestra.run.trial020.stdout:(37/42): python3-portend-3.1.0-2.el9.noarch.rpm 4.2 MB/s | 16 kB 00:00 2026-02-24T15:41:56.013 INFO:teuthology.orchestra.run.trial020.stdout:(38/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 26 MB/s | 90 kB 00:00 2026-02-24T15:41:56.017 INFO:teuthology.orchestra.run.trial020.stdout:(39/42): python3-tempora-5.0.0-2.el9.noarch.rpm 9.5 MB/s | 36 kB 00:00 2026-02-24T15:41:56.020 INFO:teuthology.orchestra.run.trial020.stdout:(40/42): python3-zc-lockfile-2.0-10.el9.noarch. 5.6 MB/s | 20 kB 00:00 2026-02-24T15:42:02.481 INFO:teuthology.orchestra.run.trial020.stdout:(41/42): ceph-osd-classic-20.3.0-5343.gd26e5c22 839 kB/s | 15 MB 00:18 2026-02-24T15:43:40.247 INFO:teuthology.orchestra.run.trial020.stdout:(42/42): ceph-osd-crimson-20.3.0-5343.gd26e5c22 778 kB/s | 88 MB 01:55 2026-02-24T15:43:40.249 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:43:40.249 INFO:teuthology.orchestra.run.trial020.stdout:Total 1.0 MB/s | 115 MB 01:58 2026-02-24T15:43:40.368 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:43:40.387 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:43:40.387 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:43:40.785 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:43:40.786 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:43:41.213 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:43:41.215 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-24T15:43:41.220 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-24T15:43:41.224 INFO:teuthology.orchestra.run.trial020.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-24T15:43:41.227 INFO:teuthology.orchestra.run.trial020.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-24T15:43:41.232 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-24T15:43:41.256 INFO:teuthology.orchestra.run.trial020.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-24T15:43:41.260 INFO:teuthology.orchestra.run.trial020.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-24T15:43:41.264 INFO:teuthology.orchestra.run.trial020.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-24T15:43:41.269 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-24T15:43:41.275 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-24T15:43:41.279 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-24T15:43:41.285 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-24T15:43:41.290 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-24T15:43:41.293 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-24T15:43:41.299 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-24T15:43:41.303 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-24T15:43:41.309 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-24T15:43:41.313 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-24T15:43:41.335 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-24T15:43:41.348 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-24T15:43:41.353 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-24T15:43:41.363 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-24T15:43:41.369 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T15:43:41.383 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-24T15:43:41.398 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-24T15:43:41.473 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-24T15:43:41.481 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-24T15:43:41.505 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-24T15:43:41.534 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-24T15:43:41.539 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-24T15:43:41.590 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-24T15:43:41.631 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 32/42 2026-02-24T15:43:41.632 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T15:43:41.655 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T15:43:41.655 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:43:41.655 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T15:43:41.656 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T15:43:41.656 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T15:43:41.656 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:41.661 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-24T15:43:41.678 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:43:41.678 INFO:teuthology.orchestra.run.trial020.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T15:43:41.678 INFO:teuthology.orchestra.run.trial020.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T15:43:41.678 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:41.686 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:43:41.708 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:43:41.708 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T15:43:41.708 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:42.306 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-24T15:43:47.708 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 37/42 2026-02-24T15:43:47.719 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 38/42 2026-02-24T15:43:47.720 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T15:43:47.743 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T15:43:47.743 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:43:47.744 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T15:43:47.744 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T15:43:47.744 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T15:43:47.744 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:47.948 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T15:43:47.968 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T15:43:47.969 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:43:47.969 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T15:43:47.969 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T15:43:47.969 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T15:43:47.969 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:48.050 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T15:43:48.069 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T15:43:48.069 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:43:48.070 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T15:43:48.070 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T15:43:48.070 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T15:43:48.070 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:48.071 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T15:43:48.077 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T15:43:48.099 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T15:43:49.061 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 7/42 2026-02-24T15:43:49.062 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 8/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-24T15:43:49.063 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-24T15:43:49.064 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-24T15:43:49.065 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T15:43:49.122 INFO:teuthology.orchestra.run.trial020.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: zip-3.0-35.el9.x86_64 2026-02-24T15:43:49.123 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:49.124 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:43:49.166 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-base 2026-02-24T15:43:49.485 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:04:09 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:43:49.550 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:43:49.568 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:43:49.569 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:43:49.569 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:43:49.591 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install cephadm 2026-02-24T15:43:49.904 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:04:09 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:43:49.986 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout: cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 1.0 M 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 1.0 M 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 1.0 M 2026-02-24T15:43:49.987 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:43:51.411 INFO:teuthology.orchestra.run.trial020.stdout:cephadm-20.3.0-5343.gd26e5c22.el9.noarch.rpm 731 kB/s | 1.0 MB 00:01 2026-02-24T15:43:51.412 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:43:51.412 INFO:teuthology.orchestra.run.trial020.stdout:Total 730 kB/s | 1.0 MB 00:01 2026-02-24T15:43:51.412 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:43:51.414 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:43:51.414 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:43:51.418 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:43:51.419 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:43:51.437 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:43:51.498 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:43:51.513 INFO:teuthology.orchestra.run.trial020.stdout: Installing : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:43:51.583 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:43:51.642 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:43:51.642 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:51.642 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:43:51.642 INFO:teuthology.orchestra.run.trial020.stdout: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:43:51.642 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:51.642 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:43:51.670 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T15:43:51.979 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:04:11 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:43:52.061 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repo Size 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 149 k 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 149 k 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 455 k 2026-02-24T15:43:52.062 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:43:53.196 INFO:teuthology.orchestra.run.trial020.stdout:ceph-immutable-object-cache-20.3.0-5343.gd26e5c 132 kB/s | 149 kB 00:01 2026-02-24T15:43:53.197 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:43:53.197 INFO:teuthology.orchestra.run.trial020.stdout:Total 132 kB/s | 149 kB 00:01 2026-02-24T15:43:53.197 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:43:53.201 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:43:53.201 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:43:53.227 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:43:53.227 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:43:53.277 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:43:53.278 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:43:53.299 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:43:53.299 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:43:53.300 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T15:43:53.300 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:53.619 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:43:53.619 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:53.619 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:43:53.619 INFO:teuthology.orchestra.run.trial020.stdout: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:53.619 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:53.619 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:43:53.645 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr 2026-02-24T15:43:53.956 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:04:13 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:43:54.020 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:43:54.038 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:43:54.039 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:43:54.039 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:43:54.059 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T15:43:54.367 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:04:14 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 51 M 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 57 k 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 17 k 2026-02-24T15:43:54.451 INFO:teuthology.orchestra.run.trial020.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout:Install 13 Packages 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 55 M 2026-02-24T15:43:54.452 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 262 M 2026-02-24T15:43:54.453 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:43:55.566 INFO:teuthology.orchestra.run.trial020.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5343.gd26 48 kB/s | 17 kB 00:00 2026-02-24T15:43:55.626 INFO:teuthology.orchestra.run.trial020.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5343.gd2 135 kB/s | 57 kB 00:00 2026-02-24T15:43:56.162 INFO:teuthology.orchestra.run.trial020.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 80 kB/s | 48 kB 00:00 2026-02-24T15:43:56.406 INFO:teuthology.orchestra.run.trial020.stdout:(4/13): python3-protobuf-3.14.0-17.el9.noarch.r 343 kB/s | 267 kB 00:00 2026-02-24T15:43:56.486 INFO:teuthology.orchestra.run.trial020.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 6.8 MB/s | 551 kB 00:00 2026-02-24T15:43:56.495 INFO:teuthology.orchestra.run.trial020.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 2.0 MB/s | 19 kB 00:00 2026-02-24T15:43:56.527 INFO:teuthology.orchestra.run.trial020.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 63 MB/s | 2.0 MB 00:00 2026-02-24T15:43:56.537 INFO:teuthology.orchestra.run.trial020.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 15 MB/s | 144 kB 00:00 2026-02-24T15:43:56.547 INFO:teuthology.orchestra.run.trial020.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 3.2 MB/s | 31 kB 00:00 2026-02-24T15:43:56.557 INFO:teuthology.orchestra.run.trial020.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 18 MB/s | 188 kB 00:00 2026-02-24T15:43:56.566 INFO:teuthology.orchestra.run.trial020.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 2.3 MB/s | 22 kB 00:00 2026-02-24T15:43:57.063 INFO:teuthology.orchestra.run.trial020.stdout:(12/13): protobuf-compiler-3.14.0-17.el9.x86_64 956 kB/s | 862 kB 00:00 2026-02-24T15:45:06.011 INFO:teuthology.orchestra.run.trial020.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5343.gd26e5c 733 kB/s | 51 MB 01:10 2026-02-24T15:45:06.013 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:06.013 INFO:teuthology.orchestra.run.trial020.stdout:Total 785 kB/s | 55 MB 01:11 2026-02-24T15:45:06.038 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:06.048 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:06.048 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:06.258 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:06.258 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:06.487 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:06.492 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-24T15:45:06.504 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-24T15:45:06.506 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-24T15:45:06.534 INFO:teuthology.orchestra.run.trial020.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-24T15:45:06.577 INFO:teuthology.orchestra.run.trial020.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-24T15:45:06.586 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-24T15:45:06.614 INFO:teuthology.orchestra.run.trial020.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-24T15:45:06.624 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-24T15:45:06.631 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-24T15:45:06.634 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-24T15:45:06.643 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 11/13 2026-02-24T15:45:09.519 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 12/13 2026-02-24T15:45:09.526 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T15:45:09.808 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T15:45:09.808 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 1/13 2026-02-24T15:45:09.808 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 2/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 3/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-24T15:45:09.809 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-24T15:45:09.810 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout: ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout: ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:09.869 INFO:teuthology.orchestra.run.trial020.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:09.870 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:09.903 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T15:45:10.217 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:05:30 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:10.301 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:10.302 INFO:teuthology.orchestra.run.trial020.stdout:======================================================================================= 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout:======================================================================================= 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 7.4 M 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T15:45:10.303 INFO:teuthology.orchestra.run.trial020.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout:======================================================================================= 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout:Install 12 Packages 2026-02-24T15:45:10.304 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:10.305 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 43 M 2026-02-24T15:45:10.305 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 234 M 2026-02-24T15:45:10.305 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:11.208 INFO:teuthology.orchestra.run.trial020.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 251 kB/s | 184 kB 00:00 2026-02-24T15:45:11.237 INFO:teuthology.orchestra.run.trial020.stdout:(2/12): flexiblas-3.0.4-9.el9.x86_64.rpm 1.0 MB/s | 30 kB 00:00 2026-02-24T15:45:11.297 INFO:teuthology.orchestra.run.trial020.stdout:(3/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 49 MB/s | 3.0 MB 00:00 2026-02-24T15:45:11.307 INFO:teuthology.orchestra.run.trial020.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 1.6 MB/s | 15 kB 00:00 2026-02-24T15:45:11.317 INFO:teuthology.orchestra.run.trial020.stdout:(5/12): openblas-0.3.29-1.el9.x86_64.rpm 4.1 MB/s | 42 kB 00:00 2026-02-24T15:45:11.358 INFO:teuthology.orchestra.run.trial020.stdout:(6/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 129 MB/s | 5.3 MB 00:00 2026-02-24T15:45:11.370 INFO:teuthology.orchestra.run.trial020.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 21 MB/s | 244 kB 00:00 2026-02-24T15:45:11.373 INFO:teuthology.orchestra.run.trial020.stdout:(8/12): libgfortran-11.5.0-14.el9.x86_64.rpm 884 kB/s | 794 kB 00:00 2026-02-24T15:45:11.416 INFO:teuthology.orchestra.run.trial020.stdout:(9/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 134 MB/s | 6.1 MB 00:00 2026-02-24T15:45:11.431 INFO:teuthology.orchestra.run.trial020.stdout:(10/12): python3-numpy-f2py-1.23.5-2.el9.x86_64 7.4 MB/s | 442 kB 00:00 2026-02-24T15:45:11.537 INFO:teuthology.orchestra.run.trial020.stdout:(11/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 160 MB/s | 19 MB 00:00 2026-02-24T15:45:19.260 INFO:teuthology.orchestra.run.trial020.stdout:(12/12): ceph-mgr-diskprediction-local-20.3.0-5 862 kB/s | 7.4 MB 00:08 2026-02-24T15:45:19.261 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:19.261 INFO:teuthology.orchestra.run.trial020.stdout:Total 4.8 MB/s | 43 MB 00:08 2026-02-24T15:45:19.416 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:19.430 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:19.430 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:19.631 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:19.632 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:19.848 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:19.851 INFO:teuthology.orchestra.run.trial020.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-24T15:45:19.857 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-24T15:45:19.891 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-24T15:45:19.895 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-24T15:45:19.955 INFO:teuthology.orchestra.run.trial020.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-24T15:45:19.957 INFO:teuthology.orchestra.run.trial020.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-24T15:45:19.978 INFO:teuthology.orchestra.run.trial020.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-24T15:45:20.297 INFO:teuthology.orchestra.run.trial020.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-24T15:45:20.369 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-24T15:45:21.029 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T15:45:21.496 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-24T15:45:21.497 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T15:45:21.778 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T15:45:21.778 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 1/12 2026-02-24T15:45:21.778 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T15:45:21.779 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-24T15:45:21.840 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-24T15:45:21.840 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T15:45:21.841 INFO:teuthology.orchestra.run.trial020.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T15:45:21.842 INFO:teuthology.orchestra.run.trial020.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T15:45:21.842 INFO:teuthology.orchestra.run.trial020.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T15:45:21.842 INFO:teuthology.orchestra.run.trial020.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T15:45:21.842 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:21.842 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:21.877 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-rook 2026-02-24T15:45:22.188 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:05:42 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:22.272 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:22.273 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:22.273 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:45:22.273 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:22.273 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:22.273 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-rook noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 50 k 2026-02-24T15:45:22.273 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:22.274 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:22.275 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:22.275 INFO:teuthology.orchestra.run.trial020.stdout:Install 10 Packages 2026-02-24T15:45:22.275 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:22.275 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 2.0 M 2026-02-24T15:45:22.275 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 25 M 2026-02-24T15:45:22.275 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:22.543 INFO:teuthology.orchestra.run.trial020.stdout:(1/10): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.7 MB/s | 157 kB 00:00 2026-02-24T15:45:22.553 INFO:teuthology.orchestra.run.trial020.stdout:(2/10): python3-pyasn1-modules-0.4.8-7.el9.noar 2.8 MB/s | 277 kB 00:00 2026-02-24T15:45:22.571 INFO:teuthology.orchestra.run.trial020.stdout:(3/10): python3-requests-oauthlib-1.3.0-12.el9. 1.9 MB/s | 54 kB 00:00 2026-02-24T15:45:22.773 INFO:teuthology.orchestra.run.trial020.stdout:(4/10): python3-certifi-2023.05.07-4.el9.noarch 70 kB/s | 14 kB 00:00 2026-02-24T15:45:22.828 INFO:teuthology.orchestra.run.trial020.stdout:(5/10): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2026-02-24T15:45:22.875 INFO:teuthology.orchestra.run.trial020.stdout:(6/10): ceph-mgr-rook-20.3.0-5343.gd26e5c22.el9 120 kB/s | 50 kB 00:00 2026-02-24T15:45:23.042 INFO:teuthology.orchestra.run.trial020.stdout:(7/10): python3-google-auth-2.45.0-1.el9.noarch 947 kB/s | 254 kB 00:00 2026-02-24T15:45:23.117 INFO:teuthology.orchestra.run.trial020.stdout:(8/10): python3-websocket-client-1.2.3-2.el9.no 1.2 MB/s | 90 kB 00:00 2026-02-24T15:45:23.186 INFO:teuthology.orchestra.run.trial020.stdout:(9/10): python3-kubernetes-26.1.0-3.el9.noarch. 2.9 MB/s | 1.0 MB 00:00 2026-02-24T15:45:23.285 INFO:teuthology.orchestra.run.trial020.stdout:(10/10): python3-rsa-4.9-2.el9.noarch.rpm 144 kB/s | 59 kB 00:00 2026-02-24T15:45:23.287 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:23.287 INFO:teuthology.orchestra.run.trial020.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2026-02-24T15:45:23.309 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:23.318 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:23.318 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:23.358 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:23.359 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:23.433 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:23.483 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 1/10 2026-02-24T15:45:23.496 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 2/10 2026-02-24T15:45:23.510 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-24T15:45:23.515 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-24T15:45:23.523 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-24T15:45:23.562 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-24T15:45:23.574 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T15:45:23.891 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-24T15:45:23.906 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-24T15:45:23.907 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 1/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 2/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 3/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-24T15:45:24.017 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-24T15:45:24.079 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-24T15:45:24.079 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:24.079 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:24.079 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:24.079 INFO:teuthology.orchestra.run.trial020.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T15:45:24.079 INFO:teuthology.orchestra.run.trial020.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T15:45:24.079 INFO:teuthology.orchestra.run.trial020.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:24.080 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:24.109 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T15:45:24.424 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:05:44 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 184 k 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout:Install 7 Packages 2026-02-24T15:45:24.509 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:24.510 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 7.1 M 2026-02-24T15:45:24.510 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 34 M 2026-02-24T15:45:24.510 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:26.002 INFO:teuthology.orchestra.run.trial020.stdout:(1/7): ceph-mgr-cephadm-20.3.0-5343.gd26e5c22.e 295 kB/s | 184 kB 00:00 2026-02-24T15:45:26.120 INFO:teuthology.orchestra.run.trial020.stdout:(2/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 335 kB/s | 249 kB 00:00 2026-02-24T15:45:26.327 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:45:26.342 INFO:teuthology.orchestra.run.trial020.stdout:(3/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.4 MB/s | 548 kB 00:00 2026-02-24T15:45:26.373 INFO:teuthology.orchestra.run.trial020.stdout:(4/7): python3-natsort-7.1.1-5.el9.noarch.rpm 1.9 MB/s | 58 kB 00:00 2026-02-24T15:45:26.403 INFO:teuthology.orchestra.run.trial020.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 2.8 MB/s | 86 kB 00:00 2026-02-24T15:45:26.430 INFO:teuthology.orchestra.run.trial020.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:45:26.492 INFO:teuthology.orchestra.run.trial020.stdout:(6/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 71 kB/s | 35 kB 00:00 2026-02-24T15:45:32.622 INFO:teuthology.orchestra.run.trial020.stdout:(7/7): python3-babel-2.9.1-2.el9.noarch.rpm 843 kB/s | 6.0 MB 00:07 2026-02-24T15:45:32.624 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:32.624 INFO:teuthology.orchestra.run.trial020.stdout:Total 895 kB/s | 7.1 MB 00:08 2026-02-24T15:45:32.661 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:32.668 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:32.668 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:32.742 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:32.742 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:32.833 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:32.882 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-24T15:45:32.891 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-24T15:45:32.897 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-24T15:45:33.116 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T15:45:33.139 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-24T15:45:33.164 INFO:teuthology.orchestra.run.trial020.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-24T15:45:33.165 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T15:45:33.276 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T15:45:33.276 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/7 2026-02-24T15:45:33.276 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-24T15:45:33.277 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-24T15:45:33.277 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T15:45:33.277 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-24T15:45:33.277 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-24T15:45:33.338 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-24T15:45:33.338 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T15:45:33.339 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:33.340 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:33.368 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-osd 2026-02-24T15:45:33.677 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:05:53 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:33.741 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:33.760 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:33.760 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:33.761 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:33.782 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-osd-classic 2026-02-24T15:45:34.093 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:05:54 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:34.160 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:34.179 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:34.179 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:34.179 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:34.200 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-fuse 2026-02-24T15:45:34.509 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:05:54 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:34.593 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:34.593 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repo Size 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout: ceph-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 914 k 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:34.594 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 914 k 2026-02-24T15:45:34.595 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 2.9 M 2026-02-24T15:45:34.595 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:38.688 INFO:teuthology.orchestra.run.trial020.stdout:ceph-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 223 kB/s | 914 kB 00:04 2026-02-24T15:45:38.689 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:38.690 INFO:teuthology.orchestra.run.trial020.stdout:Total 223 kB/s | 914 kB 00:04 2026-02-24T15:45:38.690 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:38.695 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:38.695 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:38.728 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:38.728 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:38.817 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:38.835 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:39.050 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:39.112 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:39.112 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:39.112 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:39.112 INFO:teuthology.orchestra.run.trial020.stdout: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:39.113 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:39.113 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:39.139 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-volume 2026-02-24T15:45:39.457 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:05:59 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:39.541 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout: ceph-volume noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 301 k 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:39.542 INFO:teuthology.orchestra.run.trial020.stdout:Install 2 Packages 2026-02-24T15:45:39.543 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:39.543 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 652 k 2026-02-24T15:45:39.543 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 2.2 M 2026-02-24T15:45:39.543 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:40.442 INFO:teuthology.orchestra.run.trial020.stdout:(1/2): ceph-volume-20.3.0-5343.gd26e5c22.el9.no 377 kB/s | 301 kB 00:00 2026-02-24T15:45:40.475 INFO:teuthology.orchestra.run.trial020.stdout:(2/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 422 kB/s | 351 kB 00:00 2026-02-24T15:45:40.476 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:40.476 INFO:teuthology.orchestra.run.trial020.stdout:Total 699 kB/s | 652 kB 00:00 2026-02-24T15:45:40.487 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:40.494 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:40.494 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:40.537 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:40.538 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:40.618 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:40.680 INFO:teuthology.orchestra.run.trial020.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-24T15:45:40.682 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T15:45:40.698 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T15:45:40.698 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:45:40.698 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T15:45:40.698 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:40.925 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 1/2 2026-02-24T15:45:40.987 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-24T15:45:40.987 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:40.988 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:40.988 INFO:teuthology.orchestra.run.trial020.stdout: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T15:45:40.988 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:40.988 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:41.015 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install librados-devel 2026-02-24T15:45:41.331 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:01 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repo Size 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout: librados-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 124 k 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:41.416 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:41.417 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:41.417 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:45:41.417 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:41.417 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 124 k 2026-02-24T15:45:41.417 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 434 k 2026-02-24T15:45:41.417 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:41.898 INFO:teuthology.orchestra.run.trial020.stdout:librados-devel-20.3.0-5343.gd26e5c22.el9.x86_64 257 kB/s | 124 kB 00:00 2026-02-24T15:45:41.898 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:41.898 INFO:teuthology.orchestra.run.trial020.stdout:Total 256 kB/s | 124 kB 00:00 2026-02-24T15:45:41.898 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:41.901 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:41.901 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:41.932 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:41.932 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:41.986 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:42.003 INFO:teuthology.orchestra.run.trial020.stdout: Installing : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:42.096 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:42.157 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:42.158 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:42.158 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:42.158 INFO:teuthology.orchestra.run.trial020.stdout: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:42.158 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:42.158 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:42.186 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install libcephfs2 2026-02-24T15:45:42.505 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:02 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:42.569 INFO:teuthology.orchestra.run.trial020.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:42.588 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:42.589 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:42.589 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:42.609 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install libcephfs-devel 2026-02-24T15:45:42.925 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:02 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:43.009 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:43.009 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:43.009 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repo Size 2026-02-24T15:45:43.009 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:43.009 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:43.009 INFO:teuthology.orchestra.run.trial020.stdout: libcephfs-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 37 k 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 26 k 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout:Install 2 Packages 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 63 k 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 227 k 2026-02-24T15:45:43.010 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:43.359 INFO:teuthology.orchestra.run.trial020.stdout:(1/2): libcephfs-devel-20.3.0-5343.gd26e5c22.el 107 kB/s | 37 kB 00:00 2026-02-24T15:45:43.361 INFO:teuthology.orchestra.run.trial020.stdout:(2/2): libcephfs-proxy2-20.3.0-5343.gd26e5c22.e 73 kB/s | 26 kB 00:00 2026-02-24T15:45:43.362 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:43.362 INFO:teuthology.orchestra.run.trial020.stdout:Total 178 kB/s | 63 kB 00:00 2026-02-24T15:45:43.363 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:43.365 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:43.365 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:43.397 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:43.397 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:43.446 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:43.446 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:45:43.461 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:45:43.478 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:45:43.536 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:45:43.536 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:45:43.598 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:45:43.598 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:43.598 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:43.598 INFO:teuthology.orchestra.run.trial020.stdout: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:43.598 INFO:teuthology.orchestra.run.trial020.stdout: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:43.598 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:43.598 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:43.625 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install librados2 2026-02-24T15:45:43.945 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:03 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:44.010 INFO:teuthology.orchestra.run.trial020.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:44.028 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:44.029 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:44.029 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:44.051 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install librbd1 2026-02-24T15:45:44.369 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:04 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:44.436 INFO:teuthology.orchestra.run.trial020.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:44.454 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:44.455 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:44.455 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:44.476 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-rados 2026-02-24T15:45:44.793 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:04 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:44.857 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:44.876 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:44.876 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:44.876 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:44.897 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-rgw 2026-02-24T15:45:45.211 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:05 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:45.276 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:45.295 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:45.295 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:45.296 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:45.317 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-cephfs 2026-02-24T15:45:45.634 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:05 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:45.700 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:45.719 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:45.720 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:45.720 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:45.741 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-rbd 2026-02-24T15:45:46.055 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:06 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:46.122 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:46.140 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:46.141 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:46.141 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:46.163 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install rbd-fuse 2026-02-24T15:45:46.480 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:06 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:46.564 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:46.564 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:46.564 INFO:teuthology.orchestra.run.trial020.stdout: Package Architecture Version Repository Size 2026-02-24T15:45:46.564 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:46.564 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:46.564 INFO:teuthology.orchestra.run.trial020.stdout: rbd-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 86 k 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 86 k 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 236 k 2026-02-24T15:45:46.565 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:47.270 INFO:teuthology.orchestra.run.trial020.stdout:rbd-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 122 kB/s | 86 kB 00:00 2026-02-24T15:45:47.272 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:47.272 INFO:teuthology.orchestra.run.trial020.stdout:Total 122 kB/s | 86 kB 00:00 2026-02-24T15:45:47.272 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:47.276 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:47.277 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:47.307 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:47.307 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:47.357 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:47.374 INFO:teuthology.orchestra.run.trial020.stdout: Installing : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:47.458 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:47.520 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:47.520 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:47.520 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:47.520 INFO:teuthology.orchestra.run.trial020.stdout: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:47.520 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:47.520 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:47.545 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install rbd-mirror 2026-02-24T15:45:47.860 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:07 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:47.944 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:47.944 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:47.944 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repo Size 2026-02-24T15:45:47.944 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:47.944 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout: rbd-mirror x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.6 M 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 2.6 M 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 11 M 2026-02-24T15:45:47.945 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:53.590 INFO:teuthology.orchestra.run.trial020.stdout:rbd-mirror-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 470 kB/s | 2.6 MB 00:05 2026-02-24T15:45:53.590 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:53.590 INFO:teuthology.orchestra.run.trial020.stdout:Total 470 kB/s | 2.6 MB 00:05 2026-02-24T15:45:53.590 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:53.596 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:53.596 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:53.634 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:53.634 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:53.796 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:53.798 INFO:teuthology.orchestra.run.trial020.stdout: Installing : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:53.820 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:53.820 INFO:teuthology.orchestra.run.trial020.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:45:53.820 INFO:teuthology.orchestra.run.trial020.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T15:45:53.820 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T15:45:53.820 INFO:teuthology.orchestra.run.trial020.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T15:45:53.820 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:54.159 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:54.159 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:54.159 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:54.159 INFO:teuthology.orchestra.run.trial020.stdout: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:54.159 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:54.160 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:54.187 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install rbd-nbd 2026-02-24T15:45:54.504 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:14 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout: Package Architecture Version Repository Size 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout: rbd-nbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 167 k 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:45:54.589 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:54.590 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 167 k 2026-02-24T15:45:54.590 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 496 k 2026-02-24T15:45:54.590 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T15:45:55.504 INFO:teuthology.orchestra.run.trial020.stdout:rbd-nbd-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 182 kB/s | 167 kB 00:00 2026-02-24T15:45:55.504 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:55.504 INFO:teuthology.orchestra.run.trial020.stdout:Total 182 kB/s | 167 kB 00:00 2026-02-24T15:45:55.504 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T15:45:55.509 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T15:45:55.509 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T15:45:55.539 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T15:45:55.539 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T15:45:55.595 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T15:45:55.612 INFO:teuthology.orchestra.run.trial020.stdout: Installing : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:55.702 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:55.765 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:45:55.765 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:55.765 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T15:45:55.765 INFO:teuthology.orchestra.run.trial020.stdout: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:55.765 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:55.766 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:55.794 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-osd-crimson 2026-02-24T15:45:56.115 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:16 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:56.181 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:56.200 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:56.201 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T15:45:56.201 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T15:45:56.223 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-24T15:45:56.538 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:06:16 ago on Tue 24 Feb 2026 03:39:40 PM UTC. 2026-02-24T15:45:56.624 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T15:45:56.624 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:56.624 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repo Size 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 433 M 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:Install 1 Package 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 433 M 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 2.6 G 2026-02-24T15:45:56.625 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T16:06:52.880 INFO:teuthology.orchestra.run.trial020.stdout:ceph-osd-crimson-debuginfo-20.3.0-5343.gd26e5c2 353 kB/s | 433 MB 20:56 2026-02-24T16:06:52.881 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:06:52.881 INFO:teuthology.orchestra.run.trial020.stdout:Total 353 kB/s | 433 MB 20:56 2026-02-24T16:06:52.881 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T16:06:52.882 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T16:06:52.882 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T16:06:54.179 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T16:06:54.179 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T16:07:17.772 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T16:07:17.789 INFO:teuthology.orchestra.run.trial020.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:07:19.695 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:07:19.757 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:07:19.757 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:07:19.757 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T16:07:19.757 INFO:teuthology.orchestra.run.trial020.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:07:19.757 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:07:19.757 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:19.826 DEBUG:teuthology.parallel:result is None 2026-02-24T16:07:19.827 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:07:19.979 DEBUG:teuthology.orchestra.run.trial020:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:07:20.001 INFO:teuthology.orchestra.run.trial020.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:07:20.001 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:07:20.002 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:07:20.004 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-24T16:07:20.005 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:07:20.005 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:07:20.065 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-24T16:07:20.066 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:07:20.066 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:07:20.130 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:07:20.192 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-24T16:07:20.193 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:07:20.193 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:07:20.256 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:07:20.319 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-24T16:07:20.320 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:07:20.320 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:07:20.383 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:07:20.447 INFO:teuthology.run_tasks:Running task pexec... 2026-02-24T16:07:20.468 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-24T16:07:20.469 DEBUG:teuthology.orchestra.run.trial020:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-24T16:07:20.489 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:07:20.489 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-24T16:07:20.539 DEBUG:teuthology.parallel:result is None 2026-02-24T16:07:20.539 INFO:teuthology.run_tasks:Running task install... 2026-02-24T16:07:20.553 DEBUG:teuthology.task.install:project ceph 2026-02-24T16:07:20.553 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T16:07:20.553 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T16:07:20.553 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T16:07:20.562 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T16:07:20.562 INFO:teuthology.task.install:extra packages: [] 2026-02-24T16:07:20.562 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T16:07:20.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:07:20.724 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T16:07:20.724 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T16:07:21.114 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:07:21.114 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:07:21.114 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:07:21.140 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-24T16:07:21.140 DEBUG:teuthology.orchestra.run.trial020:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:07:21.204 DEBUG:teuthology.orchestra.run.trial020:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:07:21.268 DEBUG:teuthology.orchestra.run.trial020:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:07:21.323 INFO:teuthology.orchestra.run.trial020.stdout:check_obsoletes = 1 2026-02-24T16:07:21.334 DEBUG:teuthology.orchestra.run.trial020:> sudo yum clean all 2026-02-24T16:07:21.493 INFO:teuthology.orchestra.run.trial020.stdout:56 files removed 2026-02-24T16:07:21.508 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-radosgw 2026-02-24T16:07:22.767 INFO:teuthology.orchestra.run.trial020.stdout:ceph packages for x86_64 83 kB/s | 92 kB 00:01 2026-02-24T16:07:23.430 INFO:teuthology.orchestra.run.trial020.stdout:ceph noarch packages 37 kB/s | 24 kB 00:00 2026-02-24T16:07:24.030 INFO:teuthology.orchestra.run.trial020.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-24T16:07:32.409 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.9 MB 00:08 2026-02-24T16:07:36.116 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 27 MB 00:03 2026-02-24T16:07:40.211 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 8.0 MB 00:01 2026-02-24T16:07:41.665 INFO:teuthology.orchestra.run.trial020.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 20 kB 00:00 2026-02-24T16:07:42.894 INFO:teuthology.orchestra.run.trial020.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-24T16:07:46.112 INFO:teuthology.orchestra.run.trial020.stdout:lab-extras 1.8 MB/s | 50 kB 00:00 2026-02-24T16:07:47.298 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:47.317 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:47.318 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:47.318 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:47.341 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-test 2026-02-24T16:07:47.658 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:01 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:47.722 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:47.741 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:47.742 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:47.742 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:47.764 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph 2026-02-24T16:07:48.082 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:48.147 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:48.165 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:48.166 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:48.166 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:48.187 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-base 2026-02-24T16:07:48.505 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:48.570 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:48.589 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:48.589 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:48.589 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:48.611 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install cephadm 2026-02-24T16:07:48.928 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:48.993 INFO:teuthology.orchestra.run.trial020.stdout:Package cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:07:49.012 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:49.013 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:49.013 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:49.034 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:07:49.349 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:49.415 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:49.434 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:49.434 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:49.434 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:49.456 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr 2026-02-24T16:07:49.770 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:49.835 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:49.854 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:49.855 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:49.855 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:49.876 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:07:50.188 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:50.254 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:07:50.272 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:50.273 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:50.273 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:50.295 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:07:50.611 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:50.676 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:07:50.695 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:50.695 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:50.695 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:50.717 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:07:51.035 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:51.099 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:07:51.118 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:51.119 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:51.119 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:51.140 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:07:51.459 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:51.525 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:07:51.544 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:51.545 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:51.545 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:51.566 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-osd 2026-02-24T16:07:51.886 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:51.952 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:51.971 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:51.972 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:51.972 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:51.993 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-osd-classic 2026-02-24T16:07:52.311 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:52.376 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:52.395 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:52.395 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:52.395 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:52.417 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-fuse 2026-02-24T16:07:52.738 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:52.804 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:52.823 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:52.823 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:52.823 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:52.845 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install ceph-volume 2026-02-24T16:07:53.154 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:53.219 INFO:teuthology.orchestra.run.trial020.stdout:Package ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:07:53.237 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:53.238 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:53.238 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:53.258 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install librados-devel 2026-02-24T16:07:53.574 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:53.639 INFO:teuthology.orchestra.run.trial020.stdout:Package librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:53.658 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:53.658 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:53.658 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:53.680 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install libcephfs2 2026-02-24T16:07:53.994 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:54.058 INFO:teuthology.orchestra.run.trial020.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:54.077 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:54.078 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:54.078 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:54.099 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install libcephfs-devel 2026-02-24T16:07:54.413 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:54.478 INFO:teuthology.orchestra.run.trial020.stdout:Package libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:54.497 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:54.497 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:54.497 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:54.519 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install librados2 2026-02-24T16:07:54.834 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:54.899 INFO:teuthology.orchestra.run.trial020.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:54.918 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:54.918 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:54.919 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:54.940 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install librbd1 2026-02-24T16:07:55.257 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:55.322 INFO:teuthology.orchestra.run.trial020.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:55.341 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:55.341 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:55.342 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:55.363 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-rados 2026-02-24T16:07:55.679 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:55.744 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:55.762 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:55.763 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:55.763 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:55.785 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-rgw 2026-02-24T16:07:56.097 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:56.161 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:56.179 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:56.180 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:56.180 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:56.202 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-cephfs 2026-02-24T16:07:56.522 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:56.588 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:56.606 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:56.607 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:56.607 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:56.629 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install python3-rbd 2026-02-24T16:07:56.946 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:57.011 INFO:teuthology.orchestra.run.trial020.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:57.029 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:57.030 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:57.030 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:57.052 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install rbd-fuse 2026-02-24T16:07:57.367 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:57.432 INFO:teuthology.orchestra.run.trial020.stdout:Package rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:57.451 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:57.451 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:57.451 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:57.473 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install rbd-mirror 2026-02-24T16:07:57.784 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:57.848 INFO:teuthology.orchestra.run.trial020.stdout:Package rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:57.866 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:57.867 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:57.867 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:57.888 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install rbd-nbd 2026-02-24T16:07:58.206 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:07:58.271 INFO:teuthology.orchestra.run.trial020.stdout:Package rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:07:58.290 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:07:58.291 INFO:teuthology.orchestra.run.trial020.stdout:Nothing to do. 2026-02-24T16:07:58.291 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:07:58.317 DEBUG:teuthology.parallel:result is None 2026-02-24T16:07:58.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:07:58.502 DEBUG:teuthology.orchestra.run.trial020:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:07:58.524 INFO:teuthology.orchestra.run.trial020.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:07:58.524 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:07:58.524 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:07:58.527 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-24T16:07:58.527 INFO:teuthology.run_tasks:Running task ceph... 2026-02-24T16:07:58.640 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-24T16:07:58.640 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 777 /var/log/ceph 2026-02-24T16:07:58.698 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-24T16:07:58.698 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-24T16:07:58.761 INFO:tasks.ceph:Creating extra log directories... 2026-02-24T16:07:58.761 DEBUG:teuthology.orchestra.run.trial020:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-24T16:07:58.835 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-24T16:07:58.835 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-24T16:07:58.835 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68172', 'branch': 'wip-liucm-shard-redesign-crimson-only', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/fio_4K_rand_rw}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '68172', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-liucm-shard-redesign-crimson-only'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'workunit': {'branch': 'wip-liucm-shard-redesign-crimson-only', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'owner': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'priority': 120, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9834, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-liucm-shard-redesign-crimson-only', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'targets': {'trial020.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmtDBaCt/yUDSVRCK9hDyJr6JZAW03lO8IPiV/PvE+qhkJc+Y4KwSfiZRdoiYMbCeTjfxm7jZcyCdNlZsBdXKY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-24_15:28:57', 'tube': 'trial', 'user': 'liucm', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-24T16:07:58.836 DEBUG:teuthology.orchestra.run.trial020:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-24T16:07:58.880 DEBUG:teuthology.orchestra.run.trial020:> sudo install -d -m0777 -- /var/run/ceph 2026-02-24T16:07:58.944 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:07:58.945 DEBUG:teuthology.orchestra.run.trial020:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:07:59.001 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:07:59.001 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_1 2026-02-24T16:07:59.057 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:07:59.057 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:07:59.057 INFO:teuthology.orchestra.run.trial020.stdout:Device: 6h/6d Inode: 908 Links: 1 2026-02-24T16:07:59.057 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:07:59.057 INFO:teuthology.orchestra.run.trial020.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:07:59.057 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 15:45:53.972981486 +0000 2026-02-24T16:07:59.058 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 15:39:14.053399399 +0000 2026-02-24T16:07:59.058 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 15:39:14.053399399 +0000 2026-02-24T16:07:59.058 INFO:teuthology.orchestra.run.trial020.stdout: Birth: 2026-02-24 15:39:14.053399399 +0000 2026-02-24T16:07:59.058 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:07:59.120 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T16:07:59.121 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T16:07:59.121 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.00017246 s, 3.0 MB/s 2026-02-24T16:07:59.122 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:07:59.179 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_2 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout:Device: 6h/6d Inode: 937 Links: 1 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 15:45:53.972981486 +0000 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 15:39:14.290399264 +0000 2026-02-24T16:07:59.235 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 15:39:14.290399264 +0000 2026-02-24T16:07:59.236 INFO:teuthology.orchestra.run.trial020.stdout: Birth: 2026-02-24 15:39:14.290399264 +0000 2026-02-24T16:07:59.236 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:07:59.297 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T16:07:59.297 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T16:07:59.297 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.000143125 s, 3.6 MB/s 2026-02-24T16:07:59.299 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:07:59.355 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_3 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout:Device: 6h/6d Inode: 967 Links: 1 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 15:45:53.972981486 +0000 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 15:39:14.522399132 +0000 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 15:39:14.522399132 +0000 2026-02-24T16:07:59.412 INFO:teuthology.orchestra.run.trial020.stdout: Birth: 2026-02-24 15:39:14.522399132 +0000 2026-02-24T16:07:59.413 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:07:59.473 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T16:07:59.474 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T16:07:59.474 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.000162661 s, 3.1 MB/s 2026-02-24T16:07:59.475 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:07:59.532 DEBUG:teuthology.orchestra.run.trial020:> stat /dev/vg_nvme/lv_4 2026-02-24T16:07:59.589 INFO:teuthology.orchestra.run.trial020.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:07:59.589 INFO:teuthology.orchestra.run.trial020.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:07:59.589 INFO:teuthology.orchestra.run.trial020.stdout:Device: 6h/6d Inode: 993 Links: 1 2026-02-24T16:07:59.589 INFO:teuthology.orchestra.run.trial020.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:07:59.589 INFO:teuthology.orchestra.run.trial020.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:07:59.589 INFO:teuthology.orchestra.run.trial020.stdout:Access: 2026-02-24 15:45:53.973981464 +0000 2026-02-24T16:07:59.589 INFO:teuthology.orchestra.run.trial020.stdout:Modify: 2026-02-24 15:39:14.748399003 +0000 2026-02-24T16:07:59.590 INFO:teuthology.orchestra.run.trial020.stdout:Change: 2026-02-24 15:39:14.748399003 +0000 2026-02-24T16:07:59.590 INFO:teuthology.orchestra.run.trial020.stdout: Birth: 2026-02-24 15:39:14.748399003 +0000 2026-02-24T16:07:59.590 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:07:59.650 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records in 2026-02-24T16:07:59.650 INFO:teuthology.orchestra.run.trial020.stderr:1+0 records out 2026-02-24T16:07:59.650 INFO:teuthology.orchestra.run.trial020.stderr:512 bytes copied, 0.000146091 s, 3.5 MB/s 2026-02-24T16:07:59.652 DEBUG:teuthology.orchestra.run.trial020:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:07:59.709 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:07:59.709 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial020.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:07:59.710 INFO:tasks.ceph:Generating config... 2026-02-24T16:07:59.712 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-24T16:07:59.712 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-24T16:07:59.712 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-24T16:07:59.712 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-24T16:07:59.712 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-24T16:07:59.712 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-24T16:07:59.713 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-24T16:07:59.714 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-24T16:07:59.714 INFO:tasks.ceph:[global] auth client required = none 2026-02-24T16:07:59.714 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-24T16:07:59.714 INFO:tasks.ceph:[global] auth service required = none 2026-02-24T16:07:59.714 INFO:tasks.ceph:[global] auth supported = none 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-24T16:07:59.715 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-24T16:07:59.716 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-24T16:07:59.717 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-24T16:07:59.718 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-24T16:07:59.719 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-24T16:07:59.720 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-24T16:07:59.720 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-24T16:07:59.720 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-24T16:07:59.720 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-24T16:07:59.720 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-24T16:07:59.720 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-24T16:07:59.720 INFO:tasks.ceph:Setting up mon.a... 2026-02-24T16:07:59.720 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-24T16:07:59.791 INFO:teuthology.orchestra.run.trial020.stdout:creating /etc/ceph/ceph.keyring 2026-02-24T16:07:59.794 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-24T16:07:59.878 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:07:59.943 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.20')] 2026-02-24T16:07:59.944 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.20', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-24T16:07:59.945 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:07:59.946 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:07:59.999 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.20 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:08:00.078 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:00.076+0000 7f491b892f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:00.078 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:00.076+0000 7f491b892f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:00.078 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:00.076+0000 7f491b892f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:00.079 INFO:teuthology.orchestra.run.trial020.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:monmaptool: generated fsid 21ae6aa4-6209-4233-a49d-4e4062dcdb97 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:setting min_mon_release = tentacle 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:epoch 0 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:fsid 21ae6aa4-6209-4233-a49d-4e4062dcdb97 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:last_changed 2026-02-24T16:08:00.077561+0000 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:created 2026-02-24T16:08:00.077561+0000 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:min_mon_release 20 (tentacle) 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:election_strategy: 1 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:0: [v2:10.20.193.20:3300/0,v1:10.20.193.20:6789/0] mon.a 2026-02-24T16:08:00.080 INFO:teuthology.orchestra.run.trial020.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-24T16:08:00.081 DEBUG:teuthology.orchestra.run.trial020:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:08:00.136 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 21ae6aa4-6209-4233-a49d-4e4062dcdb97... 2026-02-24T16:08:00.138 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-24T16:08:00.216 INFO:teuthology.orchestra.run.trial020.stdout:[global] 2026-02-24T16:08:00.216 INFO:teuthology.orchestra.run.trial020.stdout: chdir = "" 2026-02-24T16:08:00.216 INFO:teuthology.orchestra.run.trial020.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: auth supported = none 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: filestore xattr use omap = true 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: mon clock drift allowed = 1.000 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: osd crush chooseleaf type = 0 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: auth debug = true 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: ms die on old message = true 2026-02-24T16:08:00.217 INFO:teuthology.orchestra.run.trial020.stdout: ms die on bug = true 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: mon pg warn max object skew = 0 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: # disable pg_autoscaler by default for new pools 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: osd pool default size = 2 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: mon osd allow primary affinity = true 2026-02-24T16:08:00.218 INFO:teuthology.orchestra.run.trial020.stdout: mon osd allow pg remap = true 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on legacy crush tunables = false 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on crush straw calc version zero = false 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on no sortbitwise = false 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on osd down out interval zero = false 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon warn on too few osds = false 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: mon_allow_pool_size_one = true 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial020.stdout: osd default data pool replay window = 5 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: mon allow pool delete = true 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: mon cluster log file level = debug 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: debug asserts on shutdown = true 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: mon health detail to clog = false 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: mon host = 10.20.193.20 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: auth client required = none 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: auth cluster required = none 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: auth service required = none 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: debug asok = 0/0 2026-02-24T16:08:00.220 INFO:teuthology.orchestra.run.trial020.stdout: debug auth = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug bdev = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug bluefs = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug bluestore = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug buffer = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug civetweb = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug client = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug compressor = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug context = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug crush = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug crypto = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug dpdk = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug eventtrace = 0/0 2026-02-24T16:08:00.221 INFO:teuthology.orchestra.run.trial020.stdout: debug filer = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug filestore = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug finisher = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug fuse = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug heartbeatmap = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug javaclient = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug journal = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug journaler = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug kstore = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug leveldb = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug lockdep = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug mds = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug mds balancer = 0/0 2026-02-24T16:08:00.222 INFO:teuthology.orchestra.run.trial020.stdout: debug mds locker = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug mds log = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug mds log expire = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug mds migrator = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug memdb = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug mgr = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug mgrc = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug mon = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug monc = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug ms = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug objclass = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug objectcacher = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug objecter = 0/0 2026-02-24T16:08:00.223 INFO:teuthology.orchestra.run.trial020.stdout: debug optracker = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug osd = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug paxos = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug perfcounter = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rados = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rbd = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rbd mirror = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rbd replay = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug refs = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rgw = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rgw lifecycle = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rgw notification = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rgw sync = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug rocksdb = 0/0 2026-02-24T16:08:00.224 INFO:teuthology.orchestra.run.trial020.stdout: debug striper = 0/0 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: debug throttle = 0/0 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: debug timer = 0/0 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: debug tp = 0/0 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: ms client mode = crc 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: ms cluster mode = crc 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: ms mon client mode = crc 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: ms mon cluster mode = crc 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: ms mon service mode = crc 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: ms service mode = crc 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: fsid = 21ae6aa4-6209-4233-a49d-4e4062dcdb97 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.225 INFO:teuthology.orchestra.run.trial020.stdout:[osd] 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd journal size = 100 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd scrub load threshold = 5.0 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd scrub max interval = 600 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd mclock profile = high_recovery_ops 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd mclock skip benchmark = true 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd recover clone overlap = true 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd recovery max chunk = 1048576 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd debug shutdown = true 2026-02-24T16:08:00.226 INFO:teuthology.orchestra.run.trial020.stdout: osd debug op order = true 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: osd debug verify stray on activate = true 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: osd debug trim objects = true 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: osd open classes on start = true 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: osd debug pg log writeout = true 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: osd deep scrub update digest min age = 30 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: osd map max advance = 10 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: journal zero on create = true 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.227 INFO:teuthology.orchestra.run.trial020.stdout: filestore ondisk finisher threads = 3 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: filestore apply finisher threads = 3 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: bdev debug aio = true 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: osd debug misdirected ops = true 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug monc = 20 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: crimson cpu num = 4 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: crimson osd obc lru size = 10 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug filestore = 0/0 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug journal = 0/0 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug ms = 0/0 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug osd = 0/0 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore = 20 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore backref = 20 2026-02-24T16:08:00.228 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore cache = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore cleaner = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore device = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore epm = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore fixedkv tree = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore journal = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore lba = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore odata = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore omap = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore onode = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore t = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: debug seastore tm = 20 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: osd objectstore = seastore 2026-02-24T16:08:00.229 INFO:teuthology.orchestra.run.trial020.stdout: osd shutdown pgref assert = True 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: seastore cachepin size pershard = 64M 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: seastore cachepin type = 2Q 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: seastore max concurrent transactions = 8 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout:[mgr] 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: debug ms = 1 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: debug mgr = 20 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: debug mon = 20 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: debug auth = 20 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:08:00.230 INFO:teuthology.orchestra.run.trial020.stdout: mgr/telemetry/nag = false 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout:[mon] 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: debug ms = 0/0 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: debug mon = 0/0 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: debug paxos = 0/0 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: debug auth = 20 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: mon data avail warn = 5 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: mon mgr mkfs grace = 240 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: mon osd reporter subtree level = osd 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: mon osd prime pg temp = true 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:08:00.231 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: auth mon ticket ttl = 660 # 11m 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: auth service ticket ttl = 240 # 4m 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: # don't complain about insecure global_id in the test suite 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: # 1m isn't quite enough 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: mon_down_mkfs_grace = 2m 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.232 INFO:teuthology.orchestra.run.trial020.stdout: mon_warn_on_filestore_osds = false 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout: osd pool default crimson = True 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout:[client] 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout: rgw cache enabled = true 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout: rgw enable ops log = true 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout: rgw enable usage log = true 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-24T16:08:00.233 INFO:teuthology.orchestra.run.trial020.stdout:[mon.a] 2026-02-24T16:08:00.234 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-24T16:08:00.235 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-24T16:08:00.323 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-24T16:08:00.323 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:00.324 DEBUG:teuthology.orchestra.run.trial020:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-24T16:08:00.340 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:00.340 DEBUG:teuthology.orchestra.run.trial020:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-24T16:08:00.397 INFO:tasks.ceph:Sending monmap to node ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:08:00.397 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:00.397 DEBUG:teuthology.orchestra.run.trial020:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-24T16:08:00.397 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:08:00.470 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:00.470 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:08:00.526 INFO:tasks.ceph:Setting up mon nodes... 2026-02-24T16:08:00.526 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-24T16:08:00.526 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:08:00.615 INFO:teuthology.orchestra.run.trial020.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:08:00.618 INFO:tasks.ceph:Setting up mds nodes... 2026-02-24T16:08:00.618 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-24T16:08:00.618 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-24T16:08:00.659 INFO:teuthology.orchestra.run.trial020.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-24T16:08:00.670 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-24T16:08:00.671 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial020.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:08:00.671 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-24T16:08:00.735 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:08:00.735 INFO:tasks.ceph:role: osd.0 2026-02-24T16:08:00.735 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:08:00.735 DEBUG:teuthology.orchestra.run.trial020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-24T16:08:00.800 INFO:teuthology.orchestra.run.trial020.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:08:00.800 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:08:00.800 INFO:teuthology.orchestra.run.trial020.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:08:00.800 INFO:teuthology.orchestra.run.trial020.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:08:00.800 INFO:teuthology.orchestra.run.trial020.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:08:00.800 INFO:teuthology.orchestra.run.trial020.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:08:00.800 INFO:teuthology.orchestra.run.trial020.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:08:00.801 INFO:teuthology.orchestra.run.trial020.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:08:00.801 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:08:00.801 INFO:teuthology.orchestra.run.trial020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:08:01.369 INFO:teuthology.orchestra.run.trial020.stdout:Discarding blocks...Done. 2026-02-24T16:08:01.371 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial020.front.sepia.ceph.com -o noatime 2026-02-24T16:08:01.371 DEBUG:teuthology.orchestra.run.trial020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-24T16:08:01.439 DEBUG:teuthology.orchestra.run.trial020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-24T16:08:01.507 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-24T16:08:01.571 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:08:01.571 INFO:tasks.ceph:role: osd.1 2026-02-24T16:08:01.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:08:01.571 DEBUG:teuthology.orchestra.run.trial020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:08:01.636 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:08:01.637 INFO:teuthology.orchestra.run.trial020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:08:02.195 INFO:teuthology.orchestra.run.trial020.stdout:Discarding blocks...Done. 2026-02-24T16:08:02.198 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial020.front.sepia.ceph.com -o noatime 2026-02-24T16:08:02.198 DEBUG:teuthology.orchestra.run.trial020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-24T16:08:02.266 DEBUG:teuthology.orchestra.run.trial020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-24T16:08:02.333 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-24T16:08:02.396 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:08:02.396 INFO:tasks.ceph:role: osd.2 2026-02-24T16:08:02.397 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:08:02.397 DEBUG:teuthology.orchestra.run.trial020:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-24T16:08:02.460 INFO:teuthology.orchestra.run.trial020.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:08:02.460 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:08:02.461 INFO:teuthology.orchestra.run.trial020.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:08:03.022 INFO:teuthology.orchestra.run.trial020.stdout:Discarding blocks...Done. 2026-02-24T16:08:03.024 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial020.front.sepia.ceph.com -o noatime 2026-02-24T16:08:03.025 DEBUG:teuthology.orchestra.run.trial020:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-24T16:08:03.096 DEBUG:teuthology.orchestra.run.trial020:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-24T16:08:03.163 DEBUG:teuthology.orchestra.run.trial020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:08:03.314 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:08:03.368 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:03,367 seastar - Reactor backend: linux-aio 2026-02-24T16:08:03.377 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:03,376 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:03.377 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:03,376 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:03.379 INFO:teuthology.orchestra.run.trial020.stderr:==39290==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:08:03.439 INFO:teuthology.orchestra.run.trial020.stderr:DEBUG 2026-02-24 16:08:03,439 [shard 0:main] seastar - reactor::drain 2026-02-24T16:08:05.513 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:05.513 INFO:teuthology.orchestra.run.trial020.stderr:Suppressions used: 2026-02-24T16:08:05.514 INFO:teuthology.orchestra.run.trial020.stderr: count bytes template 2026-02-24T16:08:05.514 INFO:teuthology.orchestra.run.trial020.stderr: 1 8 MallocExtension::Register 2026-02-24T16:08:05.514 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:05.514 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:05.534 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:08:05.534 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:08:05.575 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:08:05.575 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:08:05.582 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:05,581 seastar - Reactor backend: linux-aio 2026-02-24T16:08:05.592 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:05,591 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:05.592 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:05,591 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:05.596 INFO:teuthology.orchestra.run.trial020.stderr:==39288==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:08:05.596 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:05,595 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:08:05.756 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xea7eb 0xd1c39c 0x9f7ac11 0xa0b4853 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.756 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc589 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x776e7c7 0x776e866 0x755bcdb 0x763e0a6 0x7efeae4 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.756 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11913d /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xec776 0xcee0e1 0x773f96f 0x75cae6c 0x74261b4 0x7bc7311 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.788 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9f2f498 0x9f2f744 0x9f2f7fd 0x9f2fceb 0x9f300ef 0x9f303ac 0x9f311f4 0xa0a9b3b 0xa0a9c35 0x7eff112 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:08:05.848 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x773f965 0x75cae6c 0x74261b4 0x7eff288 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.849 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15829 0x10d1d95 0xf741d0 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5af8 0x9fec307 0x9fec9e7 0xa0d221c 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.849 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0xa0c2c45 0xa0c2f8a 0xa0c330d 0xa0d3464 0xa0d68c7 0xa096e39 0xa095e39 0x7f00012 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.962 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1190fb /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xef905 0x3422b88 0x3422d62 0x3423019 0x342310e 0x7242ba8 0x7eff2ec 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.962 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191ccd /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x776e7c7 0x801a0bc 0x7ee8e78 0x7eff6a2 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.962 INFO:teuthology.orchestra.run.trial020.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8d90dce9 0xffffffff8d5d1df7 0xffffffff8d2910a4 0xffffffff8df50451 0xffffffff8e000ba2 2026-02-24T16:08:05.962 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0x9f2f2d6 0x9f2f3ec 0x9f2f4c8 0x9f2fb84 0x9f2fceb 0x9f300ef 0x9f303ac 0xa0d4a95 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:05.962 INFO:teuthology.orchestra.run.trial020.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8d29519a 0xffffffff8d5e7269 0xffffffff8d5ca7d5 0xffffffff8d5d4bab 0xffffffff8d5d500b 0xffffffff8d2910d2 0xffffffff8df50451 0xffffffff8e000ba2 2026-02-24T16:08:08.147 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:08.147 INFO:teuthology.orchestra.run.trial020.stderr:Suppressions used: 2026-02-24T16:08:08.147 INFO:teuthology.orchestra.run.trial020.stderr: count bytes template 2026-02-24T16:08:08.147 INFO:teuthology.orchestra.run.trial020.stderr: 1 8 MallocExtension::Register 2026-02-24T16:08:08.148 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:08.148 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:08.183 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-24T16:08:08.207 DEBUG:teuthology.orchestra.run.trial020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:08:08.361 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:08:08.416 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:08,416 seastar - Reactor backend: linux-aio 2026-02-24T16:08:08.434 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:08,433 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:08.434 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:08,433 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:08.436 INFO:teuthology.orchestra.run.trial020.stderr:==39356==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:08:08.496 INFO:teuthology.orchestra.run.trial020.stderr:DEBUG 2026-02-24 16:08:08,495 [shard 0:main] seastar - reactor::drain 2026-02-24T16:08:10.559 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:10.559 INFO:teuthology.orchestra.run.trial020.stderr:Suppressions used: 2026-02-24T16:08:10.559 INFO:teuthology.orchestra.run.trial020.stderr: count bytes template 2026-02-24T16:08:10.559 INFO:teuthology.orchestra.run.trial020.stderr: 1 8 MallocExtension::Register 2026-02-24T16:08:10.559 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:10.559 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:10.579 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:08:10.579 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:08:10.621 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:08:10.621 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:08:10.628 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:10,627 seastar - Reactor backend: linux-aio 2026-02-24T16:08:10.644 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:10,643 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:10.644 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:10,643 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:10.648 INFO:teuthology.orchestra.run.trial020.stderr:==39354==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:08:10.648 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:10,647 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:08:10.806 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc4a7 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0x7ebcf88 0xa0d8806 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:10.807 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xe838 /lib64/libasan.so.8+0xef905 0x3422b88 0x3422d62 0x3423019 0x342310e 0x7242ba8 0x7eff2ec 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:10.807 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191ccd /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x776e7c7 0x776e866 0x755bcdb 0x763e0a6 0x7efeae4 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:10.807 INFO:teuthology.orchestra.run.trial020.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8d6621ff 0xffffffff8d663603 0xffffffff8d5cc8ed 0xffffffff8d5d4bab 0xffffffff8d5d500b 0xffffffff8d2910d2 0xffffffff8df50451 0xffffffff8e000ba2 2026-02-24T16:08:10.839 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc7e7 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xa0d87b8 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:08:10.866 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xe6ca62 0xa0d8782 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:10.898 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x341fb22 0x9fe526f 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:10.899 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x4f8d5a8 0x9ff18fd 0xa0b4147 0xa0b430f 0xa0b4470 0xa0b4964 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:11.012 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0xa0c339c 0xa0d3464 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:11.013 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xa28f /lib64/libasan.so.8+0xe8e0 /lib64/libasan.so.8+0xec776 0xcee0e1 0x773f96f 0x75cae6c 0x74261b4 0x7eff354 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:11.013 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xebf0 /lib64/libasan.so.8+0xec776 0xcee0e1 0x773f994 0x75cae6c 0x74261b4 0x7eff354 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:13.201 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:13.202 INFO:teuthology.orchestra.run.trial020.stderr:Suppressions used: 2026-02-24T16:08:13.202 INFO:teuthology.orchestra.run.trial020.stderr: count bytes template 2026-02-24T16:08:13.202 INFO:teuthology.orchestra.run.trial020.stderr: 1 8 MallocExtension::Register 2026-02-24T16:08:13.202 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:13.202 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:13.237 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-24T16:08:13.262 DEBUG:teuthology.orchestra.run.trial020:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:08:13.413 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:08:13.469 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:13,469 seastar - Reactor backend: linux-aio 2026-02-24T16:08:13.477 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:13,476 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:13.477 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:13,476 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:13.479 INFO:teuthology.orchestra.run.trial020.stderr:==39425==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:08:13.539 INFO:teuthology.orchestra.run.trial020.stderr:DEBUG 2026-02-24 16:08:13,539 [shard 0:main] seastar - reactor::drain 2026-02-24T16:08:15.564 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:15.565 INFO:teuthology.orchestra.run.trial020.stderr:Suppressions used: 2026-02-24T16:08:15.565 INFO:teuthology.orchestra.run.trial020.stderr: count bytes template 2026-02-24T16:08:15.565 INFO:teuthology.orchestra.run.trial020.stderr: 1 8 MallocExtension::Register 2026-02-24T16:08:15.565 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:15.565 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:15.585 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:08:15.585 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:08:15.627 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:08:15.627 INFO:teuthology.orchestra.run.trial020.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:08:15.634 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:15,633 seastar - Reactor backend: linux-aio 2026-02-24T16:08:15.644 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:15,643 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:15.644 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:15,643 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:08:15.648 INFO:teuthology.orchestra.run.trial020.stderr:==39423==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:08:15.648 INFO:teuthology.orchestra.run.trial020.stderr:INFO 2026-02-24 16:08:15,647 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:08:15.806 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb06e4 /lib64/libasan.so.8+0xb1437 0xba66a7 0x10d1d1e 0xf741d0 0xa0ce5c9 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:15.806 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xea798 0xe6c75a 0xe6cb15 0xe6cd7e 0x77b1595 0x77b1696 0x763e27c 0x7efeae4 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:15.806 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10b6c3 0xe6c75a 0xe6cb15 0xa0d8782 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:15.838 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xa0d87af 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:08:15.898 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9fe5281 0x9fe5537 0x9fe5af8 0x9fec307 0x9fec9e7 0xa0d221c 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:15.898 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x157c7 0x10d1d95 0xf741d0 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe6fa7 0x9fe74f4 0x9fe76f3 0xa0c7a65 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:15.900 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xbab3c3 0xa0d6ad1 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:16.016 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc249 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0xa09aab6 0xa09abe4 0xa0cdb7b 0xa0ce31e 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:16.017 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10d1d95 0xf741d0 0x9fe4b44 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:16.017 INFO:teuthology.orchestra.run.trial020.stderr:Reactor stalled for 33 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1294d6 /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x9fe4bad 0x9fe500e 0x9fe5281 0x9fe6fa7 0xa0c5c7a 0xa0c7620 0xa0c7c7a 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:08:18.165 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:18.166 INFO:teuthology.orchestra.run.trial020.stderr:Suppressions used: 2026-02-24T16:08:18.166 INFO:teuthology.orchestra.run.trial020.stderr: count bytes template 2026-02-24T16:08:18.166 INFO:teuthology.orchestra.run.trial020.stderr: 1 8 MallocExtension::Register 2026-02-24T16:08:18.166 INFO:teuthology.orchestra.run.trial020.stderr:----------------------------------------------------- 2026-02-24T16:08:18.166 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:18.201 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-24T16:08:18.227 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-24T16:08:18.227 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:18.228 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-24T16:08:18.291 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:18.291 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-24T16:08:18.355 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:18.355 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-24T16:08:18.418 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:18.418 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-24T16:08:18.482 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:18.483 DEBUG:teuthology.orchestra.run.trial020:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-24T16:08:18.538 INFO:tasks.ceph:Adding keys to all mons... 2026-02-24T16:08:18.539 DEBUG:teuthology.orchestra.run.trial020:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-24T16:08:18.600 INFO:teuthology.orchestra.run.trial020.stdout:[mgr.x] 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout: key = AQDgzJ1piGugJBAAKbkyeDWwVmAVPna0nksMHA== 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout:[osd.0] 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout: key = AQDlzJ1pzlJ0JxAAQ7Xlk2k81xgOWe5dHYiYQg== 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout:[osd.1] 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout: key = AQDqzJ1pli96KhAAigH38teAGkAosokbnt6PAQ== 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout:[osd.2] 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout: key = AQDvzJ1pM7l7KhAACUJ7cwl8Tbx+pKf5QpdC+g== 2026-02-24T16:08:18.601 INFO:teuthology.orchestra.run.trial020.stdout:[client.0] 2026-02-24T16:08:18.602 INFO:teuthology.orchestra.run.trial020.stdout: key = AQDgzJ1pYP1DJxAAi0GJZtkAXw7Ha8MwM2f4pg== 2026-02-24T16:08:18.603 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-24T16:08:18.689 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:08:18.737 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:08:18.785 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:08:18.833 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-24T16:08:18.883 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-24T16:08:18.883 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-24T16:08:18.907 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-24T16:08:18.993 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:18.991+0000 7f0d3fc5ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:18.993 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:18.991+0000 7f0d3fc5ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.000 DEBUG:teuthology.orchestra.run.trial020:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-24T16:08:19.025 DEBUG:teuthology.orchestra.run.trial020:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:08:19.080 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-24T16:08:19.081 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-24T16:08:19.081 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-24T16:08:19.123 INFO:tasks.ceph.mon.a:Started 2026-02-24T16:08:19.123 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-24T16:08:19.123 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-24T16:08:19.123 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-24T16:08:19.125 INFO:tasks.ceph.mgr.x:Started 2026-02-24T16:08:19.126 DEBUG:tasks.ceph:set 0 configs 2026-02-24T16:08:19.126 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph config dump 2026-02-24T16:08:19.186 INFO:tasks.ceph.mon.a.trial020.stderr:2026-02-24T16:08:19.184+0000 7f8b13632c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.186 INFO:tasks.ceph.mon.a.trial020.stderr:2026-02-24T16:08:19.184+0000 7f8b13632c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.186 INFO:tasks.ceph.mon.a.trial020.stderr:2026-02-24T16:08:19.184+0000 7f8b13632c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.187 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T16:08:19.186+0000 7f550b42ffc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.200 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:19.197+0000 7f71a319c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.204 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:19.202+0000 7f71a319c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.235 INFO:teuthology.orchestra.run.trial020.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-24T16:08:19.243 INFO:tasks.ceph:Setting crush tunables to default 2026-02-24T16:08:19.243 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd crush tunables default 2026-02-24T16:08:19.317 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:19.312+0000 7ff97dc1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.318 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:19.316+0000 7ff97dc1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.350 INFO:teuthology.orchestra.run.trial020.stderr:adjusted tunables profile to default 2026-02-24T16:08:19.358 INFO:tasks.ceph:check_enable_crimson: True 2026-02-24T16:08:19.358 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-24T16:08:19.358 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-24T16:08:19.391 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T16:08:19.389+0000 7f550b42ffc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.391 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T16:08:19.390+0000 7f550b42ffc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.431 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:19.427+0000 7f4747374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.433 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:19.431+0000 7f4747374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:19.470 INFO:tasks.ceph.mon.a.trial020.stderr:2026-02-24T16:08:19.469+0000 7f8b0a9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-24T16:08:19.471 INFO:tasks.ceph.mon.a.trial020.stderr:Please be aware that this feature is experimental, untested, 2026-02-24T16:08:19.471 INFO:tasks.ceph.mon.a.trial020.stderr:unsupported, and may result in data corruption, data loss, 2026-02-24T16:08:19.471 INFO:tasks.ceph.mon.a.trial020.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-24T16:08:19.471 INFO:tasks.ceph.mon.a.trial020.stderr:feature with important data. 2026-02-24T16:08:19.471 INFO:tasks.ceph.mon.a.trial020.stderr: 2026-02-24T16:08:20.224 INFO:tasks.ceph.mgr.x.trial020.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T16:08:20.224 INFO:tasks.ceph.mgr.x.trial020.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T16:08:20.224 INFO:tasks.ceph.mgr.x.trial020.stderr: from numpy import show_config as show_numpy_config 2026-02-24T16:08:20.361 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-24T16:08:20.361 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:20.361 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-24T16:08:20.424 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:20.424 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-24T16:08:20.486 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:20.487 DEBUG:teuthology.orchestra.run.trial020:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-24T16:08:20.551 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd new 0c1e6125-ba71-4b51-b2b0-d99c237c6180 0 2026-02-24T16:08:20.664 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:20.659+0000 7f420c793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:20.665 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:20.663+0000 7f420c793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:20.702 INFO:teuthology.orchestra.run.trial020.stdout:0 2026-02-24T16:08:20.711 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd new ef63a994-4bd0-4745-9250-0842c279942c 1 2026-02-24T16:08:20.784 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:20.779+0000 7f292b98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:20.785 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:20.783+0000 7f292b98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:20.823 INFO:teuthology.orchestra.run.trial020.stdout:1 2026-02-24T16:08:20.831 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd new f02591ce-d616-4060-9106-59d132817c32 2 2026-02-24T16:08:20.906 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:20.901+0000 7fce7bf0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:20.907 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:20.906+0000 7fce7bf0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:20.943 INFO:teuthology.orchestra.run.trial020.stdout:2 2026-02-24T16:08:20.952 DEBUG:teuthology.orchestra.run.trial020:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-24T16:08:20.980 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-24T16:08:20.980 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-24T16:08:21.022 INFO:tasks.ceph.osd.0:Started 2026-02-24T16:08:21.022 DEBUG:teuthology.orchestra.run.trial020:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-24T16:08:21.051 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-24T16:08:21.051 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-24T16:08:21.094 INFO:tasks.ceph.osd.1:Started 2026-02-24T16:08:21.094 DEBUG:teuthology.orchestra.run.trial020:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-24T16:08:21.120 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-24T16:08:21.121 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-24T16:08:21.163 INFO:tasks.ceph.osd.2:Started 2026-02-24T16:08:21.163 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:08:21.238 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:21.235+0000 7f11687dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:21.240 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:21.238+0000 7f11687dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:21.270 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:21.270 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":6,"fsid":"21ae6aa4-6209-4233-a49d-4e4062dcdb97","created":"2026-02-24T16:08:19.199470+0000","modified":"2026-02-24T16:08:20.942524+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:08:20.942524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"0c1e6125-ba71-4b51-b2b0-d99c237c6180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ef63a994-4bd0-4745-9250-0842c279942c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f02591ce-d616-4060-9106-59d132817c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:08:21.278 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-24T16:08:21.278 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-24T16:08:21.580 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:08:21.648 INFO:teuthology.misc.health.trial020.stderr:2026-02-24T16:08:21.644+0000 7fb603046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:21.649 INFO:teuthology.misc.health.trial020.stderr:2026-02-24T16:08:21.647+0000 7fb603046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:21.680 INFO:teuthology.misc.health.trial020.stdout: 2026-02-24T16:08:21.681 INFO:teuthology.misc.health.trial020.stdout:{"epoch":6,"fsid":"21ae6aa4-6209-4233-a49d-4e4062dcdb97","created":"2026-02-24T16:08:19.199470+0000","modified":"2026-02-24T16:08:20.942524+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:08:20.942524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"0c1e6125-ba71-4b51-b2b0-d99c237c6180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ef63a994-4bd0-4745-9250-0842c279942c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f02591ce-d616-4060-9106-59d132817c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:08:21.688 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:08:22.452 INFO:tasks.ceph.mgr.x.trial020.stderr:2026-02-24T16:08:22.449+0000 7f542802c640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-24T16:08:27.990 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:08:28.059 INFO:teuthology.misc.health.trial020.stderr:2026-02-24T16:08:28.056+0000 7f059da46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:28.061 INFO:teuthology.misc.health.trial020.stderr:2026-02-24T16:08:28.059+0000 7f059da46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:28.170 INFO:teuthology.misc.health.trial020.stdout: 2026-02-24T16:08:28.170 INFO:teuthology.misc.health.trial020.stdout:{"epoch":8,"fsid":"21ae6aa4-6209-4233-a49d-4e4062dcdb97","created":"2026-02-24T16:08:19.199470+0000","modified":"2026-02-24T16:08:27.731444+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:08:20.942524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"0c1e6125-ba71-4b51-b2b0-d99c237c6180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ef63a994-4bd0-4745-9250-0842c279942c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f02591ce-d616-4060-9106-59d132817c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:08:28.178 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:08:34.481 DEBUG:teuthology.orchestra.run.trial020:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:08:34.548 INFO:teuthology.misc.health.trial020.stderr:2026-02-24T16:08:34.545+0000 7f06a808b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:34.549 INFO:teuthology.misc.health.trial020.stderr:2026-02-24T16:08:34.547+0000 7f06a808b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:34.659 INFO:teuthology.misc.health.trial020.stdout: 2026-02-24T16:08:34.659 INFO:teuthology.misc.health.trial020.stdout:{"epoch":13,"fsid":"21ae6aa4-6209-4233-a49d-4e4062dcdb97","created":"2026-02-24T16:08:19.199470+0000","modified":"2026-02-24T16:08:33.747776+0000","last_up_change":"2026-02-24T16:08:29.737411+0000","last_in_change":"2026-02-24T16:08:20.942524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:08:31.460988+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0c1e6125-ba71-4b51-b2b0-d99c237c6180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6803","nonce":1362392549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6802","nonce":1362392549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6811","nonce":1362392549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6812","nonce":1362392549}]},"public_addr":"10.20.193.20:6803/1362392549","cluster_addr":"10.20.193.20:6802/1362392549","heartbeat_back_addr":"10.20.193.20:6811/1362392549","heartbeat_front_addr":"10.20.193.20:6812/1362392549","state":["exists","up"]},{"osd":1,"uuid":"ef63a994-4bd0-4745-9250-0842c279942c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6804","nonce":2621078780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6805","nonce":2621078780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6813","nonce":2621078780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6810","nonce":2621078780}]},"public_addr":"10.20.193.20:6804/2621078780","cluster_addr":"10.20.193.20:6805/2621078780","heartbeat_back_addr":"10.20.193.20:6813/2621078780","heartbeat_front_addr":"10.20.193.20:6810/2621078780","state":["exists","up"]},{"osd":2,"uuid":"f02591ce-d616-4060-9106-59d132817c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6806","nonce":807694110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6807","nonce":807694110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6809","nonce":807694110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6814","nonce":807694110}]},"public_addr":"10.20.193.20:6806/807694110","cluster_addr":"10.20.193.20:6807/807694110","heartbeat_back_addr":"10.20.193.20:6809/807694110","heartbeat_front_addr":"10.20.193.20:6814/807694110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:08:34.667 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-24T16:08:34.667 INFO:tasks.ceph:Creating RBD pool 2026-02-24T16:08:34.667 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-24T16:08:34.742 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:34.737+0000 7f4a8c08b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:34.743 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:34.742+0000 7f4a8c08b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:35.757 INFO:teuthology.orchestra.run.trial020.stderr:pool 'rbd' created 2026-02-24T16:08:35.770 DEBUG:teuthology.orchestra.run.trial020:> rbd --cluster ceph pool init rbd 2026-02-24T16:08:35.812 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:35.810+0000 7f55c4cf1240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:35.812 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:35.810+0000 7f55c4cf1240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:35.813 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:35.812+0000 7f55c4cf1240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:38.791 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-24T16:08:38.791 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-24T16:08:38.792 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-24T16:08:38.909 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:38.905+0000 7f227a50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:38.910 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:38.909+0000 7f227a50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.021 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:39.031 INFO:teuthology.orchestra.run.trial020.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-24T16:08:39.031 INFO:tasks.ceph_manager:config epoch is 1 2026-02-24T16:08:39.031 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-24T16:08:39.031 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-24T16:08:39.031 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-24T16:08:39.106 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.103+0000 7faaeb65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.108 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.106+0000 7faaeb65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.248 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:39.256 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6800","nonce":854401479},{"type":"v1","addr":"10.20.193.20:6801","nonce":854401479}]},"active_addr":"10.20.193.20:6801/854401479","active_change":"2026-02-24T16:08:21.446479+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.20:0","nonce":1474244664}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.20:0","nonce":50305910}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.20:0","nonce":2267024480}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.20:0","nonce":1797655517}]}]} 2026-02-24T16:08:39.258 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-24T16:08:39.258 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-24T16:08:39.258 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:08:39.334 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.330+0000 7f938128b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.335 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.334+0000 7f938128b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.445 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:39.445 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":17,"fsid":"21ae6aa4-6209-4233-a49d-4e4062dcdb97","created":"2026-02-24T16:08:19.199470+0000","modified":"2026-02-24T16:08:38.764178+0000","last_up_change":"2026-02-24T16:08:29.737411+0000","last_in_change":"2026-02-24T16:08:20.942524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:08:31.460988+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:08:34.857449+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0c1e6125-ba71-4b51-b2b0-d99c237c6180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6803","nonce":1362392549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6802","nonce":1362392549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6811","nonce":1362392549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6812","nonce":1362392549}]},"public_addr":"10.20.193.20:6803/1362392549","cluster_addr":"10.20.193.20:6802/1362392549","heartbeat_back_addr":"10.20.193.20:6811/1362392549","heartbeat_front_addr":"10.20.193.20:6812/1362392549","state":["exists","up"]},{"osd":1,"uuid":"ef63a994-4bd0-4745-9250-0842c279942c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6804","nonce":2621078780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6805","nonce":2621078780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6813","nonce":2621078780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6810","nonce":2621078780}]},"public_addr":"10.20.193.20:6804/2621078780","cluster_addr":"10.20.193.20:6805/2621078780","heartbeat_back_addr":"10.20.193.20:6813/2621078780","heartbeat_front_addr":"10.20.193.20:6810/2621078780","state":["exists","up"]},{"osd":2,"uuid":"f02591ce-d616-4060-9106-59d132817c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6806","nonce":807694110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6807","nonce":807694110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6809","nonce":807694110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6814","nonce":807694110}]},"public_addr":"10.20.193.20:6806/807694110","cluster_addr":"10.20.193.20:6807/807694110","heartbeat_back_addr":"10.20.193.20:6809/807694110","heartbeat_front_addr":"10.20.193.20:6814/807694110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:08:39.456 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-24T16:08:39.456 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:08:39.531 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.527+0000 7ff391835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.532 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.531+0000 7ff391835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.641 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:39.642 INFO:teuthology.orchestra.run.trial020.stdout:{"epoch":17,"fsid":"21ae6aa4-6209-4233-a49d-4e4062dcdb97","created":"2026-02-24T16:08:19.199470+0000","modified":"2026-02-24T16:08:38.764178+0000","last_up_change":"2026-02-24T16:08:29.737411+0000","last_in_change":"2026-02-24T16:08:20.942524+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:08:31.460988+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:08:34.857449+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0c1e6125-ba71-4b51-b2b0-d99c237c6180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6803","nonce":1362392549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6802","nonce":1362392549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6811","nonce":1362392549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6812","nonce":1362392549}]},"public_addr":"10.20.193.20:6803/1362392549","cluster_addr":"10.20.193.20:6802/1362392549","heartbeat_back_addr":"10.20.193.20:6811/1362392549","heartbeat_front_addr":"10.20.193.20:6812/1362392549","state":["exists","up"]},{"osd":1,"uuid":"ef63a994-4bd0-4745-9250-0842c279942c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6804","nonce":2621078780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6805","nonce":2621078780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6813","nonce":2621078780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6810","nonce":2621078780}]},"public_addr":"10.20.193.20:6804/2621078780","cluster_addr":"10.20.193.20:6805/2621078780","heartbeat_back_addr":"10.20.193.20:6813/2621078780","heartbeat_front_addr":"10.20.193.20:6810/2621078780","state":["exists","up"]},{"osd":2,"uuid":"f02591ce-d616-4060-9106-59d132817c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6806","nonce":807694110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6807","nonce":807694110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6809","nonce":807694110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.20:6814","nonce":807694110}]},"public_addr":"10.20.193.20:6806/807694110","cluster_addr":"10.20.193.20:6807/807694110","heartbeat_back_addr":"10.20.193.20:6809/807694110","heartbeat_front_addr":"10.20.193.20:6814/807694110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:08:39.652 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-24T16:08:39.652 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-24T16:08:39.653 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-24T16:08:39.732 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.727+0000 7fb97b9a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.732 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.728+0000 7fbd60fb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.733 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.732+0000 7fb97b9a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.734 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.732+0000 7fbd60fb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.738 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.734+0000 7f4a567fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.739 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.738+0000 7f4a567fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.748 INFO:teuthology.orchestra.run.trial020.stdout:38654705674 2026-02-24T16:08:39.749 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T16:08:39.749 INFO:teuthology.orchestra.run.trial020.stdout:42949672969 2026-02-24T16:08:39.750 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T16:08:39.754 INFO:teuthology.orchestra.run.trial020.stdout:42949672969 2026-02-24T16:08:39.755 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T16:08:39.871 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.867+0000 7f5cbf362640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.872 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.867+0000 7f1eaa70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.873 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.871+0000 7f5cbf362640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.873 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.871+0000 7f1eaa70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.912 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.907+0000 7faf1dbb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.913 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:39.911+0000 7faf1dbb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:39.984 INFO:teuthology.orchestra.run.trial020.stdout:42949672969 2026-02-24T16:08:39.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.1 2026-02-24T16:08:39.995 DEBUG:teuthology.parallel:result is None 2026-02-24T16:08:39.996 INFO:teuthology.orchestra.run.trial020.stdout:38654705674 2026-02-24T16:08:40.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2026-02-24T16:08:40.007 DEBUG:teuthology.parallel:result is None 2026-02-24T16:08:40.029 INFO:teuthology.orchestra.run.trial020.stdout:42949672969 2026-02-24T16:08:40.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-24T16:08:40.039 DEBUG:teuthology.parallel:result is None 2026-02-24T16:08:40.040 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T16:08:40.040 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:08:40.155 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:40.151+0000 7f7d01993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:40.156 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:40.154+0000 7f7d01993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:40.264 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:40.265 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T16:08:40.273 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-24T16:08:39.747138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917004,"kb_used_data":62917004,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369348,"statfs":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590379,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":80,"num_write_kb":1107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.980746"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.776701+0000","last_change":"2026-02-24T16:08:38.780782+0000","last_active":"2026-02-24T16:08:38.776701+0000","last_peered":"2026-02-24T16:08:38.776701+0000","last_clean":"2026-02-24T16:08:38.776701+0000","last_became_active":"2026-02-24T16:08:36.798785+0000","last_became_peered":"2026-02-24T16:08:36.798785+0000","last_unstale":"2026-02-24T16:08:38.776701+0000","last_undegraded":"2026-02-24T16:08:38.776701+0000","last_fullsized":"2026-02-24T16:08:38.776701+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.779222+0000","last_change":"2026-02-24T16:08:38.782751+0000","last_active":"2026-02-24T16:08:38.779222+0000","last_peered":"2026-02-24T16:08:38.779222+0000","last_clean":"2026-02-24T16:08:38.779222+0000","last_became_active":"2026-02-24T16:08:36.796227+0000","last_became_peered":"2026-02-24T16:08:36.796227+0000","last_unstale":"2026-02-24T16:08:38.779222+0000","last_undegraded":"2026-02-24T16:08:38.779222+0000","last_fullsized":"2026-02-24T16:08:38.779222+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.778934+0000","last_change":"2026-02-24T16:08:38.784089+0000","last_active":"2026-02-24T16:08:38.778934+0000","last_peered":"2026-02-24T16:08:38.778934+0000","last_clean":"2026-02-24T16:08:38.778934+0000","last_became_active":"2026-02-24T16:08:36.797108+0000","last_became_peered":"2026-02-24T16:08:36.797108+0000","last_unstale":"2026-02-24T16:08:38.778934+0000","last_undegraded":"2026-02-24T16:08:38.778934+0000","last_fullsized":"2026-02-24T16:08:38.778934+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.778952+0000","last_change":"2026-02-24T16:08:38.782422+0000","last_active":"2026-02-24T16:08:38.778952+0000","last_peered":"2026-02-24T16:08:38.778952+0000","last_clean":"2026-02-24T16:08:38.778952+0000","last_became_active":"2026-02-24T16:08:36.797364+0000","last_became_peered":"2026-02-24T16:08:36.797364+0000","last_unstale":"2026-02-24T16:08:38.778952+0000","last_undegraded":"2026-02-24T16:08:38.778952+0000","last_fullsized":"2026-02-24T16:08:38.778952+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.778271+0000","last_change":"2026-02-24T16:08:38.778271+0000","last_active":"2026-02-24T16:08:38.778271+0000","last_peered":"2026-02-24T16:08:38.778271+0000","last_clean":"2026-02-24T16:08:38.778271+0000","last_became_active":"2026-02-24T16:08:36.808059+0000","last_became_peered":"2026-02-24T16:08:36.808059+0000","last_unstale":"2026-02-24T16:08:38.778271+0000","last_undegraded":"2026-02-24T16:08:38.778271+0000","last_fullsized":"2026-02-24T16:08:38.778271+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.776053+0000","last_change":"2026-02-24T16:08:38.778784+0000","last_active":"2026-02-24T16:08:38.776053+0000","last_peered":"2026-02-24T16:08:38.776053+0000","last_clean":"2026-02-24T16:08:38.776053+0000","last_became_active":"2026-02-24T16:08:36.804820+0000","last_became_peered":"2026-02-24T16:08:36.804820+0000","last_unstale":"2026-02-24T16:08:38.776053+0000","last_undegraded":"2026-02-24T16:08:38.776053+0000","last_fullsized":"2026-02-24T16:08:38.776053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.774555+0000","last_change":"2026-02-24T16:08:38.777356+0000","last_active":"2026-02-24T16:08:38.774555+0000","last_peered":"2026-02-24T16:08:38.774555+0000","last_clean":"2026-02-24T16:08:38.774555+0000","last_became_active":"2026-02-24T16:08:36.802144+0000","last_became_peered":"2026-02-24T16:08:36.802144+0000","last_unstale":"2026-02-24T16:08:38.774555+0000","last_undegraded":"2026-02-24T16:08:38.774555+0000","last_fullsized":"2026-02-24T16:08:38.774555+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.779070+0000","last_change":"2026-02-24T16:08:38.783992+0000","last_active":"2026-02-24T16:08:38.779070+0000","last_peered":"2026-02-24T16:08:38.779070+0000","last_clean":"2026-02-24T16:08:38.779070+0000","last_became_active":"2026-02-24T16:08:36.794704+0000","last_became_peered":"2026-02-24T16:08:36.794704+0000","last_unstale":"2026-02-24T16:08:38.779070+0000","last_undegraded":"2026-02-24T16:08:38.779070+0000","last_fullsized":"2026-02-24T16:08:38.779070+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.777515+0000","last_change":"2026-02-24T16:08:32.777430+0000","last_active":"2026-02-24T16:08:38.777515+0000","last_peered":"2026-02-24T16:08:38.777515+0000","last_clean":"2026-02-24T16:08:38.777515+0000","last_became_active":"2026-02-24T16:08:32.770477+0000","last_became_peered":"2026-02-24T16:08:32.770477+0000","last_unstale":"2026-02-24T16:08:38.777515+0000","last_undegraded":"2026-02-24T16:08:38.777515+0000","last_fullsized":"2026-02-24T16:08:38.777515+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:31.739072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:31.739072+0000","last_clean_scrub_stamp":"2026-02-24T16:08:31.739072+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972620,"kb_used_data":20972620,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456164,"statfs":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972572,"kb_used_data":20972572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456212,"statfs":{"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:08:40.275 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:08:40.351 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:40.347+0000 7f1f00c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:40.352 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:40.350+0000 7f1f00c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:40.459 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:40.460 INFO:teuthology.orchestra.run.trial020.stderr:dumped all 2026-02-24T16:08:40.468 INFO:teuthology.orchestra.run.trial020.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-24T16:08:39.747138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917004,"kb_used_data":62917004,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369348,"statfs":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590379,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":80,"num_write_kb":1107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.980746"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.776701+0000","last_change":"2026-02-24T16:08:38.780782+0000","last_active":"2026-02-24T16:08:38.776701+0000","last_peered":"2026-02-24T16:08:38.776701+0000","last_clean":"2026-02-24T16:08:38.776701+0000","last_became_active":"2026-02-24T16:08:36.798785+0000","last_became_peered":"2026-02-24T16:08:36.798785+0000","last_unstale":"2026-02-24T16:08:38.776701+0000","last_undegraded":"2026-02-24T16:08:38.776701+0000","last_fullsized":"2026-02-24T16:08:38.776701+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.779222+0000","last_change":"2026-02-24T16:08:38.782751+0000","last_active":"2026-02-24T16:08:38.779222+0000","last_peered":"2026-02-24T16:08:38.779222+0000","last_clean":"2026-02-24T16:08:38.779222+0000","last_became_active":"2026-02-24T16:08:36.796227+0000","last_became_peered":"2026-02-24T16:08:36.796227+0000","last_unstale":"2026-02-24T16:08:38.779222+0000","last_undegraded":"2026-02-24T16:08:38.779222+0000","last_fullsized":"2026-02-24T16:08:38.779222+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.778934+0000","last_change":"2026-02-24T16:08:38.784089+0000","last_active":"2026-02-24T16:08:38.778934+0000","last_peered":"2026-02-24T16:08:38.778934+0000","last_clean":"2026-02-24T16:08:38.778934+0000","last_became_active":"2026-02-24T16:08:36.797108+0000","last_became_peered":"2026-02-24T16:08:36.797108+0000","last_unstale":"2026-02-24T16:08:38.778934+0000","last_undegraded":"2026-02-24T16:08:38.778934+0000","last_fullsized":"2026-02-24T16:08:38.778934+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.778952+0000","last_change":"2026-02-24T16:08:38.782422+0000","last_active":"2026-02-24T16:08:38.778952+0000","last_peered":"2026-02-24T16:08:38.778952+0000","last_clean":"2026-02-24T16:08:38.778952+0000","last_became_active":"2026-02-24T16:08:36.797364+0000","last_became_peered":"2026-02-24T16:08:36.797364+0000","last_unstale":"2026-02-24T16:08:38.778952+0000","last_undegraded":"2026-02-24T16:08:38.778952+0000","last_fullsized":"2026-02-24T16:08:38.778952+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.778271+0000","last_change":"2026-02-24T16:08:38.778271+0000","last_active":"2026-02-24T16:08:38.778271+0000","last_peered":"2026-02-24T16:08:38.778271+0000","last_clean":"2026-02-24T16:08:38.778271+0000","last_became_active":"2026-02-24T16:08:36.808059+0000","last_became_peered":"2026-02-24T16:08:36.808059+0000","last_unstale":"2026-02-24T16:08:38.778271+0000","last_undegraded":"2026-02-24T16:08:38.778271+0000","last_fullsized":"2026-02-24T16:08:38.778271+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.776053+0000","last_change":"2026-02-24T16:08:38.778784+0000","last_active":"2026-02-24T16:08:38.776053+0000","last_peered":"2026-02-24T16:08:38.776053+0000","last_clean":"2026-02-24T16:08:38.776053+0000","last_became_active":"2026-02-24T16:08:36.804820+0000","last_became_peered":"2026-02-24T16:08:36.804820+0000","last_unstale":"2026-02-24T16:08:38.776053+0000","last_undegraded":"2026-02-24T16:08:38.776053+0000","last_fullsized":"2026-02-24T16:08:38.776053+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.774555+0000","last_change":"2026-02-24T16:08:38.777356+0000","last_active":"2026-02-24T16:08:38.774555+0000","last_peered":"2026-02-24T16:08:38.774555+0000","last_clean":"2026-02-24T16:08:38.774555+0000","last_became_active":"2026-02-24T16:08:36.802144+0000","last_became_peered":"2026-02-24T16:08:36.802144+0000","last_unstale":"2026-02-24T16:08:38.774555+0000","last_undegraded":"2026-02-24T16:08:38.774555+0000","last_fullsized":"2026-02-24T16:08:38.774555+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.779070+0000","last_change":"2026-02-24T16:08:38.783992+0000","last_active":"2026-02-24T16:08:38.779070+0000","last_peered":"2026-02-24T16:08:38.779070+0000","last_clean":"2026-02-24T16:08:38.779070+0000","last_became_active":"2026-02-24T16:08:36.794704+0000","last_became_peered":"2026-02-24T16:08:36.794704+0000","last_unstale":"2026-02-24T16:08:38.779070+0000","last_undegraded":"2026-02-24T16:08:38.779070+0000","last_fullsized":"2026-02-24T16:08:38.779070+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:35.752237+0000","last_clean_scrub_stamp":"2026-02-24T16:08:35.752237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:08:38.777515+0000","last_change":"2026-02-24T16:08:32.777430+0000","last_active":"2026-02-24T16:08:38.777515+0000","last_peered":"2026-02-24T16:08:38.777515+0000","last_clean":"2026-02-24T16:08:38.777515+0000","last_became_active":"2026-02-24T16:08:32.770477+0000","last_became_peered":"2026-02-24T16:08:32.770477+0000","last_unstale":"2026-02-24T16:08:38.777515+0000","last_undegraded":"2026-02-24T16:08:38.777515+0000","last_fullsized":"2026-02-24T16:08:38.777515+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:08:31.739072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:08:31.739072+0000","last_clean_scrub_stamp":"2026-02-24T16:08:31.739072+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972620,"kb_used_data":20972620,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456164,"statfs":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972572,"kb_used_data":20972572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456212,"statfs":{"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:08:40.469 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T16:08:40.469 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-24T16:08:40.469 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-24T16:08:40.470 DEBUG:teuthology.orchestra.run.trial020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-24T16:08:40.545 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:40.541+0000 7f2dca21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:40.546 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:08:40.545+0000 7f2dca21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:08:40.685 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:40.686 INFO:teuthology.orchestra.run.trial020.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-24T16:08:40.695 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-24T16:08:40.695 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-24T16:08:40.778 DEBUG:teuthology.orchestra.run.trial020:> echo 'StrictHostKeyChecking no 2026-02-24T16:08:40.778 DEBUG:teuthology.orchestra.run.trial020:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-24T16:08:40.794 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQC3M83UE4ra6/QP/cxFjKCZwxH0dnmCKKZZi02wfnq75EXuFj0SAcl4baDBxoTWaR/lNwicdqMeHHBVlYP/hX9E0+UVSQvW+oH67tciytleTSVzr2t24r5EHP5iLYr8jBt1EOkBDpuTH9EFUxkygqyl++pmvoUnKPOPL+BG9rmLyHBm3KgWYLjpYHxqsYTNUqmMnA7aO2QK5MAQFoSVHb6Z5Ij5IynSn/2RqQ7GrP8qC+X6iPVNtzc/VQuA6KbtR31YfBOvkQiNHzl+SWnmZQ7unVO2PRGxLuRubVY4cqguroCRetV/YVsU8pYH+UykuUus0lnDmrNFJJOZugQn+EXr 2026-02-24T16:08:40.794 INFO:teuthology.task.ssh_keys:pushing keys to trial020.front.sepia.ceph.com for ubuntu 2026-02-24T16:08:40.794 DEBUG:teuthology.orchestra.run.trial020:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:08:40.850 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:40.850 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-24T16:08:40.850 DEBUG:teuthology.orchestra.run.trial020:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-24T16:08:40.906 DEBUG:teuthology.orchestra.run.trial020:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:08:40.961 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:40.961 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:08:41.018 DEBUG:teuthology.orchestra.run.trial020:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:08:41:018193 2026-02-24T16:08:41.074 INFO:teuthology.orchestra.run.trial020.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-24_16:08:41:018193' 2026-02-24T16:08:41.075 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:41.075 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-24T16:08:41.131 INFO:teuthology.run_tasks:Running task cbt... 2026-02-24T16:08:41.155 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial020.front.sepia.ceph.com', 'osds': ['ubuntu@trial020.front.sepia.ceph.com'], 'mons': ['ubuntu@trial020.front.sepia.ceph.com'], 'clients': ['ubuntu@trial020.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial020.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-24T16:08:41.156 DEBUG:teuthology.orchestra.run.trial020:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-24T16:08:41.191 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:08:41.191 DEBUG:teuthology.orchestra.run.trial020:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:08:41.242 DEBUG:teuthology.orchestra.run.trial020:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-24T16:08:41.304 INFO:teuthology.orchestra.run.trial020.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-24T16:08:41.635 INFO:tasks.cbt:Installing collectl 2026-02-24T16:08:41.635 DEBUG:teuthology.orchestra.run.trial020:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-24T16:08:41.709 INFO:teuthology.orchestra.run.trial020.stderr:--2026-02-24 16:08:41-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-24T16:08:41.764 INFO:teuthology.orchestra.run.trial020.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, ... 2026-02-24T16:08:41.767 INFO:teuthology.orchestra.run.trial020.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.12.149|:443... connected. 2026-02-24T16:08:42.962 INFO:teuthology.orchestra.run.trial020.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:08:42.963 INFO:teuthology.orchestra.run.trial020.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnc0KFTlJG7u5CDW9_1GVCSdthIqaFbtVa2OWzLh-6cHxfGr-P3FpyOSpEHjRdES2OHJqIRoF42DLSsHjZiCOQ6VWXA%3D%3D&use_mirror=cfhcable&r= [following] 2026-02-24T16:08:42.963 INFO:teuthology.orchestra.run.trial020.stderr:--2026-02-24 16:08:42-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnc0KFTlJG7u5CDW9_1GVCSdthIqaFbtVa2OWzLh-6cHxfGr-P3FpyOSpEHjRdES2OHJqIRoF42DLSsHjZiCOQ6VWXA%3D%3D&use_mirror=cfhcable&r= 2026-02-24T16:08:43.016 INFO:teuthology.orchestra.run.trial020.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:c95, ... 2026-02-24T16:08:43.019 INFO:teuthology.orchestra.run.trial020.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.13.149|:443... connected. 2026-02-24T16:08:43.270 INFO:teuthology.orchestra.run.trial020.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:08:43.271 INFO:teuthology.orchestra.run.trial020.stderr:Location: https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-24T16:08:43.271 INFO:teuthology.orchestra.run.trial020.stderr:--2026-02-24 16:08:43-- https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-24T16:08:43.277 INFO:teuthology.orchestra.run.trial020.stderr:Resolving cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)... 146.71.73.6 2026-02-24T16:08:43.318 INFO:teuthology.orchestra.run.trial020.stderr:Connecting to cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)|146.71.73.6|:443... connected. 2026-02-24T16:08:43.525 INFO:teuthology.orchestra.run.trial020.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-24T16:08:43.525 INFO:teuthology.orchestra.run.trial020.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-24T16:08:43.526 INFO:teuthology.orchestra.run.trial020.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-24T16:08:43.526 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:43.575 INFO:teuthology.orchestra.run.trial020.stderr: 0K .......... .......... .......... .......... .......... 8% 1002K 1s 2026-02-24T16:08:43.606 INFO:teuthology.orchestra.run.trial020.stderr: 50K .......... .......... .......... .......... .......... 16% 1.54M 0s 2026-02-24T16:08:43.617 INFO:teuthology.orchestra.run.trial020.stderr: 100K .......... .......... .......... .......... .......... 24% 5.25M 0s 2026-02-24T16:08:43.648 INFO:teuthology.orchestra.run.trial020.stderr: 150K .......... .......... .......... .......... .......... 32% 1.55M 0s 2026-02-24T16:08:43.660 INFO:teuthology.orchestra.run.trial020.stderr: 200K .......... .......... .......... .......... .......... 41% 5.39M 0s 2026-02-24T16:08:43.660 INFO:teuthology.orchestra.run.trial020.stderr: 250K .......... .......... .......... .......... .......... 49% 359M 0s 2026-02-24T16:08:43.660 INFO:teuthology.orchestra.run.trial020.stderr: 300K .......... .......... .......... .......... .......... 57% 348M 0s 2026-02-24T16:08:43.660 INFO:teuthology.orchestra.run.trial020.stderr: 350K .......... .......... .......... .......... .......... 65% 340M 0s 2026-02-24T16:08:43.689 INFO:teuthology.orchestra.run.trial020.stderr: 400K .......... .......... .......... .......... .......... 74% 1.56M 0s 2026-02-24T16:08:43.699 INFO:teuthology.orchestra.run.trial020.stderr: 450K .......... .......... .......... .......... .......... 82% 5.47M 0s 2026-02-24T16:08:43.699 INFO:teuthology.orchestra.run.trial020.stderr: 500K .......... .......... .......... .......... .......... 90% 351M 0s 2026-02-24T16:08:43.699 INFO:teuthology.orchestra.run.trial020.stderr: 550K .......... .......... .......... .......... .......... 98% 201M 0s 2026-02-24T16:08:43.700 INFO:teuthology.orchestra.run.trial020.stderr: 600K ...... 100% 11.4T=0.2s 2026-02-24T16:08:43.700 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:43.700 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24 16:08:43 (3.43 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-24T16:08:43.700 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:08:43.723 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/ 2026-02-24T16:08:43.723 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/collectl.conf 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/ 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Socket.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/colmux.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Import.html 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/style.css 2026-02-24T16:08:43.724 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Process.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-24T16:08:43.725 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Export.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Startup.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Performance.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Messages.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Logging.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-24T16:08:43.726 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Misc.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Examples.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Data.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-24T16:08:43.727 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Playback.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Hello.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Network.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Memory.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/Disks.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/service/ 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/service/collectl.service 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/initd/ 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-24T16:08:43.728 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/initd/collectl 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/misc.ph 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/gexpr.ph 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/statsd.ph 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/envrules.std 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/hello.ph 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/vmsum.ph 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/proctree.ph 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/README-WINDOWS 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/formatit.ph 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/colmux 2026-02-24T16:08:43.729 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/INSTALL 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/graphite.ph 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/vmstat.ph 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/README 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/man1/ 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/man1/collectl.1 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/man1/colmux.1 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/lexpr.ph 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/client.pl 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/collectl 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/COPYING 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/vnet.ph 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/GPL 2026-02-24T16:08:43.730 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/UNINSTALL 2026-02-24T16:08:43.731 INFO:teuthology.orchestra.run.trial020.stdout:collectl-4.3.1/ARTISTIC 2026-02-24T16:08:43.765 INFO:teuthology.orchestra.run.trial020.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-24T16:08:43.767 DEBUG:teuthology.orchestra.run.trial020:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-24T16:08:44.123 INFO:teuthology.orchestra.run.trial020.stdout:Last metadata expiration check: 0:00:58 ago on Tue 24 Feb 2026 04:07:46 PM UTC. 2026-02-24T16:08:44.209 INFO:teuthology.orchestra.run.trial020.stdout:Dependencies resolved. 2026-02-24T16:08:44.210 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: Package Arch Version Repository Size 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout:Installing: 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: librbd-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 24 k 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout:Installing dependencies: 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: libradospp-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 31 k 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:44.211 INFO:teuthology.orchestra.run.trial020.stdout:Transaction Summary 2026-02-24T16:08:44.212 INFO:teuthology.orchestra.run.trial020.stdout:================================================================================ 2026-02-24T16:08:44.212 INFO:teuthology.orchestra.run.trial020.stdout:Install 6 Packages 2026-02-24T16:08:44.212 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:44.212 INFO:teuthology.orchestra.run.trial020.stdout:Total download size: 3.8 M 2026-02-24T16:08:44.212 INFO:teuthology.orchestra.run.trial020.stdout:Installed size: 14 M 2026-02-24T16:08:44.212 INFO:teuthology.orchestra.run.trial020.stdout:Downloading Packages: 2026-02-24T16:08:44.553 INFO:teuthology.orchestra.run.trial020.stdout:(1/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 4.2 MB/s | 290 kB 00:00 2026-02-24T16:08:44.838 INFO:teuthology.orchestra.run.trial020.stdout:(2/6): librbd-devel-20.3.0-5343.gd26e5c22.el9.x 67 kB/s | 24 kB 00:00 2026-02-24T16:08:44.851 INFO:teuthology.orchestra.run.trial020.stdout:(3/6): perf-5.14.0-686.el9.x86_64.rpm 11 MB/s | 3.3 MB 00:00 2026-02-24T16:08:45.043 INFO:teuthology.orchestra.run.trial020.stdout:(4/6): pdsh-2.34-7.el9.x86_64.rpm 578 kB/s | 118 kB 00:00 2026-02-24T16:08:45.058 INFO:teuthology.orchestra.run.trial020.stdout:(5/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 63 kB/s | 13 kB 00:00 2026-02-24T16:08:45.190 INFO:teuthology.orchestra.run.trial020.stdout:(6/6): libradospp-devel-20.3.0-5343.gd26e5c22.e 44 kB/s | 31 kB 00:00 2026-02-24T16:08:45.194 INFO:teuthology.orchestra.run.trial020.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:08:45.195 INFO:teuthology.orchestra.run.trial020.stdout:Total 3.8 MB/s | 3.8 MB 00:00 2026-02-24T16:08:45.218 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction check 2026-02-24T16:08:45.228 INFO:teuthology.orchestra.run.trial020.stdout:Transaction check succeeded. 2026-02-24T16:08:45.228 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction test 2026-02-24T16:08:45.284 INFO:teuthology.orchestra.run.trial020.stdout:Transaction test succeeded. 2026-02-24T16:08:45.284 INFO:teuthology.orchestra.run.trial020.stdout:Running transaction 2026-02-24T16:08:45.359 INFO:teuthology.orchestra.run.trial020.stdout: Preparing : 1/1 2026-02-24T16:08:45.364 INFO:teuthology.orchestra.run.trial020.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-24T16:08:45.404 INFO:teuthology.orchestra.run.trial020.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-24T16:08:45.408 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:08:45.410 INFO:teuthology.orchestra.run.trial020.stdout: Installing : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/6 2026-02-24T16:08:45.572 INFO:teuthology.orchestra.run.trial020.stdout: Installing : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/6 2026-02-24T16:08:45.593 INFO:teuthology.orchestra.run.trial020.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:08:45.703 INFO:teuthology.orchestra.run.trial020.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:08:45.703 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/6 2026-02-24T16:08:45.703 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/6 2026-02-24T16:08:45.704 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:08:45.704 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-24T16:08:45.704 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-24T16:08:45.766 INFO:teuthology.orchestra.run.trial020.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-24T16:08:45.766 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:45.766 INFO:teuthology.orchestra.run.trial020.stdout:Installed: 2026-02-24T16:08:45.766 INFO:teuthology.orchestra.run.trial020.stdout: libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:08:45.767 INFO:teuthology.orchestra.run.trial020.stdout: librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:08:45.767 INFO:teuthology.orchestra.run.trial020.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-24T16:08:45.767 INFO:teuthology.orchestra.run.trial020.stdout: pdsh-2.34-7.el9.x86_64 2026-02-24T16:08:45.767 INFO:teuthology.orchestra.run.trial020.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-24T16:08:45.767 INFO:teuthology.orchestra.run.trial020.stdout: perf-5.14.0-686.el9.x86_64 2026-02-24T16:08:45.767 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:45.767 INFO:teuthology.orchestra.run.trial020.stdout:Complete! 2026-02-24T16:08:45.797 DEBUG:teuthology.orchestra.run.trial020:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-24T16:08:45.936 INFO:teuthology.orchestra.run.trial020.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-24T16:08:45.948 INFO:teuthology.orchestra.run.trial020.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-24T16:08:46.414 INFO:teuthology.orchestra.run.trial020.stdout:Collecting lxml 2026-02-24T16:08:46.438 INFO:teuthology.orchestra.run.trial020.stdout: Downloading lxml-6.0.2-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.3 MB) 2026-02-24T16:08:46.809 INFO:teuthology.orchestra.run.trial020.stdout:Collecting matplotlib 2026-02-24T16:08:46.813 INFO:teuthology.orchestra.run.trial020.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-24T16:08:46.857 INFO:teuthology.orchestra.run.trial020.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-24T16:08:47.179 INFO:teuthology.orchestra.run.trial020.stdout:Collecting fonttools>=4.22.0 2026-02-24T16:08:47.184 INFO:teuthology.orchestra.run.trial020.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-24T16:08:47.241 INFO:teuthology.orchestra.run.trial020.stdout:Collecting importlib-resources>=3.2.0 2026-02-24T16:08:47.245 INFO:teuthology.orchestra.run.trial020.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-24T16:08:47.388 INFO:teuthology.orchestra.run.trial020.stdout:Collecting contourpy>=1.0.1 2026-02-24T16:08:47.392 INFO:teuthology.orchestra.run.trial020.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-24T16:08:47.395 INFO:teuthology.orchestra.run.trial020.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-24T16:08:47.395 INFO:teuthology.orchestra.run.trial020.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-24T16:08:47.551 INFO:teuthology.orchestra.run.trial020.stdout:Collecting kiwisolver>=1.3.1 2026-02-24T16:08:47.555 INFO:teuthology.orchestra.run.trial020.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-24T16:08:47.576 INFO:teuthology.orchestra.run.trial020.stdout:Collecting cycler>=0.10 2026-02-24T16:08:47.581 INFO:teuthology.orchestra.run.trial020.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-24T16:08:47.958 INFO:teuthology.orchestra.run.trial020.stdout:Collecting pillow>=8 2026-02-24T16:08:47.966 INFO:teuthology.orchestra.run.trial020.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-24T16:08:48.014 INFO:teuthology.orchestra.run.trial020.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-24T16:08:48.088 INFO:teuthology.orchestra.run.trial020.stdout:Collecting zipp>=3.1.0 2026-02-24T16:08:48.092 INFO:teuthology.orchestra.run.trial020.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-24T16:08:48.103 INFO:teuthology.orchestra.run.trial020.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-24T16:08:48.216 INFO:teuthology.orchestra.run.trial020.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib, lxml 2026-02-24T16:08:49.346 INFO:teuthology.orchestra.run.trial020.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 lxml-6.0.2 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-24T16:08:49.475 INFO:tasks.cbt:benchmark: librbdfio 2026-02-24T16:08:49.475 DEBUG:teuthology.orchestra.run.trial020:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2026-02-24T16:08:49.493 INFO:teuthology.orchestra.run.trial020.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2026-02-24T16:08:50.289 DEBUG:teuthology.orchestra.run.trial020:> cd /home/ubuntu/cephtest/fio && ./configure && make 2026-02-24T16:08:50.455 INFO:teuthology.orchestra.run.trial020.stdout:Operating system Linux 2026-02-24T16:08:50.456 INFO:teuthology.orchestra.run.trial020.stdout:CPU x86_64 2026-02-24T16:08:50.456 INFO:teuthology.orchestra.run.trial020.stdout:Big endian no 2026-02-24T16:08:50.456 INFO:teuthology.orchestra.run.trial020.stdout:Compiler gcc 2026-02-24T16:08:50.456 INFO:teuthology.orchestra.run.trial020.stdout:Cross compile no 2026-02-24T16:08:50.456 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:08:50.456 INFO:teuthology.orchestra.run.trial020.stdout:Static build no 2026-02-24T16:08:50.503 INFO:teuthology.orchestra.run.trial020.stdout:Wordsize 64 2026-02-24T16:08:50.508 INFO:teuthology.orchestra.run.trial020.stdout:zlib no 2026-02-24T16:08:50.520 INFO:teuthology.orchestra.run.trial020.stdout:fcntl(F_FULLFSYNC) no 2026-02-24T16:08:50.531 INFO:teuthology.orchestra.run.trial020.stdout:Linux AIO support no 2026-02-24T16:08:50.531 INFO:teuthology.orchestra.run.trial020.stdout:Linux AIO support rw flags no 2026-02-24T16:08:50.554 INFO:teuthology.orchestra.run.trial020.stdout:POSIX AIO support yes 2026-02-24T16:08:50.554 INFO:teuthology.orchestra.run.trial020.stdout:POSIX AIO support needs -lrt no 2026-02-24T16:08:50.575 INFO:teuthology.orchestra.run.trial020.stdout:POSIX AIO fsync yes 2026-02-24T16:08:50.595 INFO:teuthology.orchestra.run.trial020.stdout:POSIX pshared support yes 2026-02-24T16:08:50.620 INFO:teuthology.orchestra.run.trial020.stdout:pthread_condattr_setclock() yes 2026-02-24T16:08:50.643 INFO:teuthology.orchestra.run.trial020.stdout:pthread_sigmask() yes 2026-02-24T16:08:50.667 INFO:teuthology.orchestra.run.trial020.stdout:pthread_getaffinity_np() yes 2026-02-24T16:08:50.674 INFO:teuthology.orchestra.run.trial020.stdout:Solaris AIO support no 2026-02-24T16:08:50.695 INFO:teuthology.orchestra.run.trial020.stdout:__sync_fetch_and_add yes 2026-02-24T16:08:50.717 INFO:teuthology.orchestra.run.trial020.stdout:__sync_synchronize yes 2026-02-24T16:08:50.737 INFO:teuthology.orchestra.run.trial020.stdout:__sync_val_compare_and_swap yes 2026-02-24T16:08:50.742 INFO:teuthology.orchestra.run.trial020.stdout:libverbs no 2026-02-24T16:08:50.750 INFO:teuthology.orchestra.run.trial020.stdout:rdmacm no 2026-02-24T16:08:50.771 INFO:teuthology.orchestra.run.trial020.stdout:asprintf() yes 2026-02-24T16:08:50.792 INFO:teuthology.orchestra.run.trial020.stdout:vasprintf() yes 2026-02-24T16:08:50.813 INFO:teuthology.orchestra.run.trial020.stdout:Linux fallocate yes 2026-02-24T16:08:50.834 INFO:teuthology.orchestra.run.trial020.stdout:POSIX fadvise yes 2026-02-24T16:08:50.855 INFO:teuthology.orchestra.run.trial020.stdout:POSIX fallocate yes 2026-02-24T16:08:50.876 INFO:teuthology.orchestra.run.trial020.stdout:sched_setaffinity(3 arg) yes 2026-02-24T16:08:50.876 INFO:teuthology.orchestra.run.trial020.stdout:sched_setaffinity(2 arg) no 2026-02-24T16:08:50.898 INFO:teuthology.orchestra.run.trial020.stdout:clock_gettime yes 2026-02-24T16:08:50.919 INFO:teuthology.orchestra.run.trial020.stdout:CLOCK_MONOTONIC yes 2026-02-24T16:08:50.940 INFO:teuthology.orchestra.run.trial020.stdout:gettimeofday yes 2026-02-24T16:08:50.961 INFO:teuthology.orchestra.run.trial020.stdout:fdatasync yes 2026-02-24T16:08:50.982 INFO:teuthology.orchestra.run.trial020.stdout:pipe() yes 2026-02-24T16:08:51.002 INFO:teuthology.orchestra.run.trial020.stdout:pipe2() yes 2026-02-24T16:08:51.023 INFO:teuthology.orchestra.run.trial020.stdout:pread() yes 2026-02-24T16:08:51.047 INFO:teuthology.orchestra.run.trial020.stdout:sync_file_range yes 2026-02-24T16:08:51.053 INFO:teuthology.orchestra.run.trial020.stdout:ASharedMemory_create no 2026-02-24T16:08:51.061 INFO:teuthology.orchestra.run.trial020.stdout:EXT4 move extent yes 2026-02-24T16:08:51.081 INFO:teuthology.orchestra.run.trial020.stdout:Linux splice(2) yes 2026-02-24T16:08:51.088 INFO:teuthology.orchestra.run.trial020.stdout:libnuma no 2026-02-24T16:08:51.109 INFO:teuthology.orchestra.run.trial020.stdout:strsep yes 2026-02-24T16:08:51.130 INFO:teuthology.orchestra.run.trial020.stdout:strcasestr yes 2026-02-24T16:08:51.139 INFO:teuthology.orchestra.run.trial020.stdout:strlcat no 2026-02-24T16:08:51.161 INFO:teuthology.orchestra.run.trial020.stdout:getopt_long_only() yes 2026-02-24T16:08:51.185 INFO:teuthology.orchestra.run.trial020.stdout:inet_aton yes 2026-02-24T16:08:51.205 INFO:teuthology.orchestra.run.trial020.stdout:socklen_t yes 2026-02-24T16:08:51.225 INFO:teuthology.orchestra.run.trial020.stdout:__thread yes 2026-02-24T16:08:51.244 INFO:teuthology.orchestra.run.trial020.stdout:RUSAGE_THREAD yes 2026-02-24T16:08:51.263 INFO:teuthology.orchestra.run.trial020.stdout:SCHED_IDLE yes 2026-02-24T16:08:51.287 INFO:teuthology.orchestra.run.trial020.stdout:TCP_NODELAY yes 2026-02-24T16:08:51.308 INFO:teuthology.orchestra.run.trial020.stdout:vsock yes 2026-02-24T16:08:51.330 INFO:teuthology.orchestra.run.trial020.stdout:Net engine window_size yes 2026-02-24T16:08:51.354 INFO:teuthology.orchestra.run.trial020.stdout:TCP_MAXSEG yes 2026-02-24T16:08:51.375 INFO:teuthology.orchestra.run.trial020.stdout:RLIMIT_MEMLOCK yes 2026-02-24T16:08:51.397 INFO:teuthology.orchestra.run.trial020.stdout:pwritev/preadv yes 2026-02-24T16:08:51.417 INFO:teuthology.orchestra.run.trial020.stdout:pwritev2/preadv2 yes 2026-02-24T16:08:51.440 INFO:teuthology.orchestra.run.trial020.stdout:IPv6 helpers yes 2026-02-24T16:08:51.452 INFO:teuthology.orchestra.run.trial020.stdout:http engine no 2026-02-24T16:08:51.506 INFO:teuthology.orchestra.run.trial020.stdout:Rados engine yes 2026-02-24T16:08:51.570 INFO:teuthology.orchestra.run.trial020.stdout:Rados Block Device engine yes 2026-02-24T16:08:51.637 INFO:teuthology.orchestra.run.trial020.stdout:rbd_poll yes 2026-02-24T16:08:51.704 INFO:teuthology.orchestra.run.trial020.stdout:rbd_invalidate_cache yes 2026-02-24T16:08:51.770 INFO:teuthology.orchestra.run.trial020.stdout:rbd_encryption_load yes 2026-02-24T16:08:51.790 INFO:teuthology.orchestra.run.trial020.stdout:setvbuf yes 2026-02-24T16:08:51.795 INFO:teuthology.orchestra.run.trial020.stdout:Gluster API engine no 2026-02-24T16:08:51.806 INFO:teuthology.orchestra.run.trial020.stdout:s390_z196_facilities no 2026-02-24T16:08:51.806 INFO:teuthology.orchestra.run.trial020.stdout:HDFS engine no 2026-02-24T16:08:51.827 INFO:teuthology.orchestra.run.trial020.stdout:MTD yes 2026-02-24T16:08:51.832 INFO:teuthology.orchestra.run.trial020.stdout:libpmem no 2026-02-24T16:08:51.832 INFO:teuthology.orchestra.run.trial020.stdout:libpmem1_5 no 2026-02-24T16:08:51.838 INFO:teuthology.orchestra.run.trial020.stdout:libpmem2 no 2026-02-24T16:08:51.838 INFO:teuthology.orchestra.run.trial020.stdout:PMDK dev-dax engine no 2026-02-24T16:08:51.838 INFO:teuthology.orchestra.run.trial020.stdout:PMDK libpmem engine no 2026-02-24T16:08:51.843 INFO:teuthology.orchestra.run.trial020.stdout:DDN's Infinite Memory Engine no 2026-02-24T16:08:51.843 INFO:teuthology.orchestra.run.trial020.stdout:iscsi engine no 2026-02-24T16:08:51.843 INFO:teuthology.orchestra.run.trial020.stdout:NBD engine no 2026-02-24T16:08:51.849 INFO:teuthology.orchestra.run.trial020.stdout:DAOS File System (dfs) Engine no 2026-02-24T16:08:51.855 INFO:teuthology.orchestra.run.trial020.stdout:NFS engine no 2026-02-24T16:08:51.856 INFO:teuthology.orchestra.run.trial020.stdout:lex/yacc for arithmetic no 2026-02-24T16:08:51.876 INFO:teuthology.orchestra.run.trial020.stdout:getmntent yes 2026-02-24T16:08:51.888 INFO:teuthology.orchestra.run.trial020.stdout:getmntinfo no 2026-02-24T16:08:51.918 INFO:teuthology.orchestra.run.trial020.stdout:Static Assert yes 2026-02-24T16:08:51.937 INFO:teuthology.orchestra.run.trial020.stdout:bool yes 2026-02-24T16:08:51.959 INFO:teuthology.orchestra.run.trial020.stdout:strndup yes 2026-02-24T16:08:51.964 INFO:teuthology.orchestra.run.trial020.stdout:Valgrind headers no 2026-02-24T16:08:51.984 INFO:teuthology.orchestra.run.trial020.stdout:Zoned block device support yes 2026-02-24T16:08:52.003 INFO:teuthology.orchestra.run.trial020.stdout:Zoned block device capacity yes 2026-02-24T16:08:52.008 INFO:teuthology.orchestra.run.trial020.stdout:libzbc engine no 2026-02-24T16:08:52.027 INFO:teuthology.orchestra.run.trial020.stdout:NVMe uring command support yes 2026-02-24T16:08:52.038 INFO:teuthology.orchestra.run.trial020.stdout:xnvme engine no 2026-02-24T16:08:52.044 INFO:teuthology.orchestra.run.trial020.stdout:isal no 2026-02-24T16:08:52.045 INFO:teuthology.orchestra.run.trial020.stdout:isal CRC64 no 2026-02-24T16:08:52.056 INFO:teuthology.orchestra.run.trial020.stdout:libblkio engine no 2026-02-24T16:08:52.056 INFO:teuthology.orchestra.run.trial020.stdout:march_armv8_a_crc_crypto no 2026-02-24T16:08:52.056 INFO:teuthology.orchestra.run.trial020.stdout:cuda no 2026-02-24T16:08:52.056 INFO:teuthology.orchestra.run.trial020.stdout:libcufile no 2026-02-24T16:08:52.077 INFO:teuthology.orchestra.run.trial020.stdout:Build march=native yes 2026-02-24T16:08:52.082 INFO:teuthology.orchestra.run.trial020.stdout:CUnit no 2026-02-24T16:08:52.102 INFO:teuthology.orchestra.run.trial020.stdout:__kernel_rwf_t yes 2026-02-24T16:08:52.120 INFO:teuthology.orchestra.run.trial020.stdout:-Wimplicit-fallthrough=2 yes 2026-02-24T16:08:52.142 INFO:teuthology.orchestra.run.trial020.stdout:-Wno-stringop-truncation yes 2026-02-24T16:08:52.163 INFO:teuthology.orchestra.run.trial020.stdout:MADV_HUGEPAGE yes 2026-02-24T16:08:52.184 INFO:teuthology.orchestra.run.trial020.stdout:gettid yes 2026-02-24T16:08:52.206 INFO:teuthology.orchestra.run.trial020.stdout:statx(2)/libc yes 2026-02-24T16:08:52.229 INFO:teuthology.orchestra.run.trial020.stdout:statx(2)/syscall yes 2026-02-24T16:08:52.230 INFO:teuthology.orchestra.run.trial020.stdout:Windows PDB generation no 2026-02-24T16:08:52.250 INFO:teuthology.orchestra.run.trial020.stdout:timerfd_create yes 2026-02-24T16:08:52.251 INFO:teuthology.orchestra.run.trial020.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2026-02-24T16:08:52.251 INFO:teuthology.orchestra.run.trial020.stdout:Lib-based ioengines dynamic no 2026-02-24T16:08:52.293 INFO:teuthology.orchestra.run.trial020.stdout:TCMalloc support yes 2026-02-24T16:08:52.294 INFO:teuthology.orchestra.run.trial020.stdout:seed_buckets 4 2026-02-24T16:08:52.340 INFO:teuthology.orchestra.run.trial020.stderr:FIO_VERSION = fio-3.41-113-g411de 2026-02-24T16:08:52.363 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crc16.o 2026-02-24T16:08:52.398 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crc32.o 2026-02-24T16:08:52.424 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crc32c-arm64.o 2026-02-24T16:08:52.467 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crc32c-intel.o 2026-02-24T16:08:52.499 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crc32c.o 2026-02-24T16:08:52.530 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crc64.o 2026-02-24T16:08:52.560 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crc7.o 2026-02-24T16:08:52.585 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/crct10dif_common.o 2026-02-24T16:08:52.610 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/fnv.o 2026-02-24T16:08:52.639 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/md5.o 2026-02-24T16:08:52.697 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/murmur3.o 2026-02-24T16:08:52.727 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/sha1.o 2026-02-24T16:08:52.829 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/sha256.o 2026-02-24T16:08:52.971 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/sha3.o 2026-02-24T16:08:53.054 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/sha512.o 2026-02-24T16:08:53.122 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/test.o 2026-02-24T16:08:53.212 INFO:teuthology.orchestra.run.trial020.stdout: CC crc/xxhash.o 2026-02-24T16:08:53.268 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/axmap.o 2026-02-24T16:08:53.339 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/bloom.o 2026-02-24T16:08:53.390 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/flist_sort.o 2026-02-24T16:08:53.436 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/gauss.o 2026-02-24T16:08:53.487 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/getrusage.o 2026-02-24T16:08:53.514 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/hweight.o 2026-02-24T16:08:53.543 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/ieee754.o 2026-02-24T16:08:53.576 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/lfsr.o 2026-02-24T16:08:53.634 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/memalign.o 2026-02-24T16:08:53.666 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/memcpy.o 2026-02-24T16:08:53.748 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/mountcheck.o 2026-02-24T16:08:53.779 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/num2str.o 2026-02-24T16:08:53.827 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/output_buffer.o 2026-02-24T16:08:53.860 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/pattern.o 2026-02-24T16:08:53.939 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/prio_tree.o 2026-02-24T16:08:54.009 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/rand.o 2026-02-24T16:08:54.084 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/rbtree.o 2026-02-24T16:08:54.154 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/strntol.o 2026-02-24T16:08:54.187 INFO:teuthology.orchestra.run.trial020.stdout: CC lib/zipf.o 2026-02-24T16:08:54.253 INFO:teuthology.orchestra.run.trial020.stdout: CC gettime.o 2026-02-24T16:08:54.405 INFO:teuthology.orchestra.run.trial020.stdout: CC ioengines.o 2026-02-24T16:08:54.543 INFO:teuthology.orchestra.run.trial020.stdout: CC init.o 2026-02-24T16:08:54.923 INFO:teuthology.orchestra.run.trial020.stdout: CC stat.o 2026-02-24T16:08:55.629 INFO:teuthology.orchestra.run.trial020.stdout: CC log.o 2026-02-24T16:08:55.702 INFO:teuthology.orchestra.run.trial020.stdout: CC time.o 2026-02-24T16:08:55.791 INFO:teuthology.orchestra.run.trial020.stdout: CC filesetup.o 2026-02-24T16:08:56.125 INFO:teuthology.orchestra.run.trial020.stdout: CC eta.o 2026-02-24T16:08:56.287 INFO:teuthology.orchestra.run.trial020.stdout: CC verify.o 2026-02-24T16:08:56.625 INFO:teuthology.orchestra.run.trial020.stdout: CC memory.o 2026-02-24T16:08:56.720 INFO:teuthology.orchestra.run.trial020.stdout: CC io_u.o 2026-02-24T16:08:57.105 INFO:teuthology.orchestra.run.trial020.stdout: CC parse.o 2026-02-24T16:08:57.326 INFO:teuthology.orchestra.run.trial020.stdout: CC fio_sem.o 2026-02-24T16:08:57.390 INFO:teuthology.orchestra.run.trial020.stdout: CC rwlock.o 2026-02-24T16:08:57.442 INFO:teuthology.orchestra.run.trial020.stdout: CC pshared.o 2026-02-24T16:08:57.485 INFO:teuthology.orchestra.run.trial020.stdout: CC options.o 2026-02-24T16:08:57.887 INFO:teuthology.orchestra.run.trial020.stdout: CC fio_shared_sem.o 2026-02-24T16:08:57.922 INFO:teuthology.orchestra.run.trial020.stdout: CC smalloc.o 2026-02-24T16:08:58.026 INFO:teuthology.orchestra.run.trial020.stdout: CC filehash.o 2026-02-24T16:08:58.112 INFO:teuthology.orchestra.run.trial020.stdout: CC profile.o 2026-02-24T16:08:58.181 INFO:teuthology.orchestra.run.trial020.stdout: CC debug.o 2026-02-24T16:08:58.210 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/cpu.o 2026-02-24T16:08:58.286 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/mmap.o 2026-02-24T16:08:58.376 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/sync.o 2026-02-24T16:08:58.497 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/null.o 2026-02-24T16:08:58.567 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/net.o 2026-02-24T16:08:58.779 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/ftruncate.o 2026-02-24T16:08:58.839 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/fileoperations.o 2026-02-24T16:08:58.931 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/exec.o 2026-02-24T16:08:59.015 INFO:teuthology.orchestra.run.trial020.stdout: CC server.o 2026-02-24T16:08:59.369 INFO:teuthology.orchestra.run.trial020.stdout: CC client.o 2026-02-24T16:08:59.647 INFO:teuthology.orchestra.run.trial020.stdout: CC iolog.o 2026-02-24T16:08:59.881 INFO:teuthology.orchestra.run.trial020.stdout: CC backend.o 2026-02-24T16:09:00.297 INFO:teuthology.orchestra.run.trial020.stdout: CC libfio.o 2026-02-24T16:09:00.397 INFO:teuthology.orchestra.run.trial020.stdout: CC flow.o 2026-02-24T16:09:00.466 INFO:teuthology.orchestra.run.trial020.stdout: CC cconv.o 2026-02-24T16:09:00.686 INFO:teuthology.orchestra.run.trial020.stdout: CC gettime-thread.o 2026-02-24T16:09:00.748 INFO:teuthology.orchestra.run.trial020.stdout: CC helpers.o 2026-02-24T16:09:00.791 INFO:teuthology.orchestra.run.trial020.stdout: CC json.o 2026-02-24T16:09:00.894 INFO:teuthology.orchestra.run.trial020.stdout: CC idletime.o 2026-02-24T16:09:01.014 INFO:teuthology.orchestra.run.trial020.stdout: CC td_error.o 2026-02-24T16:09:01.075 INFO:teuthology.orchestra.run.trial020.stdout: CC profiles/tiobench.o 2026-02-24T16:09:01.133 INFO:teuthology.orchestra.run.trial020.stdout: CC profiles/act.o 2026-02-24T16:09:01.222 INFO:teuthology.orchestra.run.trial020.stdout: CC io_u_queue.o 2026-02-24T16:09:01.256 INFO:teuthology.orchestra.run.trial020.stdout: CC filelock.o 2026-02-24T16:09:01.328 INFO:teuthology.orchestra.run.trial020.stdout: CC workqueue.o 2026-02-24T16:09:01.424 INFO:teuthology.orchestra.run.trial020.stdout: CC rate-submit.o 2026-02-24T16:09:01.507 INFO:teuthology.orchestra.run.trial020.stdout: CC optgroup.o 2026-02-24T16:09:01.540 INFO:teuthology.orchestra.run.trial020.stdout: CC helper_thread.o 2026-02-24T16:09:01.639 INFO:teuthology.orchestra.run.trial020.stdout: CC steadystate.o 2026-02-24T16:09:01.785 INFO:teuthology.orchestra.run.trial020.stdout: CC zone-dist.o 2026-02-24T16:09:01.861 INFO:teuthology.orchestra.run.trial020.stdout: CC zbd.o 2026-02-24T16:09:02.222 INFO:teuthology.orchestra.run.trial020.stdout: CC dedupe.o 2026-02-24T16:09:02.304 INFO:teuthology.orchestra.run.trial020.stdout: CC dataplacement.o 2026-02-24T16:09:02.398 INFO:teuthology.orchestra.run.trial020.stdout: CC sprandom.o 2026-02-24T16:09:02.559 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/posixaio.o 2026-02-24T16:09:02.637 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/falloc.o 2026-02-24T16:09:02.705 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/e4defrag.o 2026-02-24T16:09:02.781 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/splice.o 2026-02-24T16:09:02.883 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/asprintf.o 2026-02-24T16:09:02.910 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/strlcat.o 2026-02-24T16:09:02.942 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/mtd.o 2026-02-24T16:09:03.029 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/libmtd.o 2026-02-24T16:09:03.221 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/libmtd_legacy.o 2026-02-24T16:09:03.294 INFO:teuthology.orchestra.run.trial020.stdout: CC diskutil.o 2026-02-24T16:09:03.409 INFO:teuthology.orchestra.run.trial020.stdout: CC fifo.o 2026-02-24T16:09:03.447 INFO:teuthology.orchestra.run.trial020.stdout: CC blktrace.o 2026-02-24T16:09:03.613 INFO:teuthology.orchestra.run.trial020.stdout: CC cgroup.o 2026-02-24T16:09:03.697 INFO:teuthology.orchestra.run.trial020.stdout: CC trim.o 2026-02-24T16:09:03.764 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/sg.o 2026-02-24T16:09:03.958 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/linux-dev-lookup.o 2026-02-24T16:09:03.995 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/io_uring.o 2026-02-24T16:09:04.228 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/nvme.o 2026-02-24T16:09:04.389 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/linux-blkzoned.o 2026-02-24T16:09:04.481 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/cmdprio.o 2026-02-24T16:09:04.606 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/rados.o 2026-02-24T16:09:04.707 INFO:teuthology.orchestra.run.trial020.stdout: CC engines/rbd.o 2026-02-24T16:09:04.813 INFO:teuthology.orchestra.run.trial020.stdout: CC fio.o 2026-02-24T16:09:04.868 INFO:teuthology.orchestra.run.trial020.stdout: LINK fio 2026-02-24T16:09:04.988 INFO:teuthology.orchestra.run.trial020.stdout: CC t/log.o 2026-02-24T16:09:05.018 INFO:teuthology.orchestra.run.trial020.stdout: CC t/genzipf.o 2026-02-24T16:09:05.092 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/strcasestr.o 2026-02-24T16:09:05.112 INFO:teuthology.orchestra.run.trial020.stdout: CC oslib/strndup.o 2026-02-24T16:09:05.132 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/fio-genzipf 2026-02-24T16:09:05.187 INFO:teuthology.orchestra.run.trial020.stdout: CC t/btrace2fio.o 2026-02-24T16:09:05.384 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/fio-btrace2fio 2026-02-24T16:09:05.437 INFO:teuthology.orchestra.run.trial020.stdout: CC t/verify-state.o 2026-02-24T16:09:05.497 INFO:teuthology.orchestra.run.trial020.stdout: CC t/debug.o 2026-02-24T16:09:05.524 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/fio-verify-state 2026-02-24T16:09:05.577 INFO:teuthology.orchestra.run.trial020.stdout: CC t/stest.o 2026-02-24T16:09:05.625 INFO:teuthology.orchestra.run.trial020.stdout: CC t/arch.o 2026-02-24T16:09:05.647 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/stest 2026-02-24T16:09:05.705 INFO:teuthology.orchestra.run.trial020.stdout: CC t/ieee754.o 2026-02-24T16:09:05.734 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/ieee754 2026-02-24T16:09:05.788 INFO:teuthology.orchestra.run.trial020.stdout: CC t/axmap.o 2026-02-24T16:09:05.846 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/axmap 2026-02-24T16:09:05.899 INFO:teuthology.orchestra.run.trial020.stdout: CC t/lfsr-test.o 2026-02-24T16:09:05.954 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/lfsr-test 2026-02-24T16:09:06.007 INFO:teuthology.orchestra.run.trial020.stdout: CC t/gen-rand.o 2026-02-24T16:09:06.062 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/gen-rand 2026-02-24T16:09:06.117 INFO:teuthology.orchestra.run.trial020.stdout: CC t/memlock.o 2026-02-24T16:09:06.157 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/memlock 2026-02-24T16:09:06.210 INFO:teuthology.orchestra.run.trial020.stdout: CC t/read-to-pipe-async.o 2026-02-24T16:09:06.311 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/read-to-pipe-async 2026-02-24T16:09:06.364 INFO:teuthology.orchestra.run.trial020.stdout: CC t/io_uring.o 2026-02-24T16:09:06.637 INFO:teuthology.orchestra.run.trial020.stdout: LINK t/io_uring 2026-02-24T16:09:06.691 DEBUG:teuthology.orchestra.run.trial020:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-24T16:09:06.718 DEBUG:teuthology.orchestra.run.trial020:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:09:06.829 INFO:teuthology.orchestra.run.trial020.stderr:16:09:06 - DEBUG - cbt - Settings.cluster: 2026-02-24T16:09:06.829 INFO:teuthology.orchestra.run.trial020.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-24T16:09:06.829 INFO:teuthology.orchestra.run.trial020.stderr: 'clients': ['ubuntu@trial020.front.sepia.ceph.com'], 2026-02-24T16:09:06.829 INFO:teuthology.orchestra.run.trial020.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-24T16:09:06.829 INFO:teuthology.orchestra.run.trial020.stderr: 'head': 'ubuntu@trial020.front.sepia.ceph.com', 2026-02-24T16:09:06.829 INFO:teuthology.orchestra.run.trial020.stderr: 'is_teuthology': True, 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'iterations': 1, 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'mons': ['ubuntu@trial020.front.sepia.ceph.com'], 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'osds': ['ubuntu@trial020.front.sepia.ceph.com'], 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'osds_per_node': 3, 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'pid_dir': '/var/run/ceph', 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'rebuild_every_test': False, 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'rgws': {'ubuntu@trial020.front.sepia.ceph.com': None}, 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'use_existing': True, 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr: 'user': 'ubuntu'} 2026-02-24T16:09:06.830 INFO:teuthology.orchestra.run.trial020.stderr:16:09:06 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-24T16:09:06.831 INFO:teuthology.orchestra.run.trial020.stderr:16:09:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw; fi 2026-02-24T16:09:06.994 INFO:teuthology.orchestra.run.trial020.stderr:16:09:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-24T16:09:07.152 INFO:teuthology.orchestra.run.trial020.stderr:16:09:07 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-24T16:09:07.152 INFO:teuthology.orchestra.run.trial020.stderr:16:09:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl 2026-02-24T16:09:07.312 INFO:teuthology.orchestra.run.trial020.stderr:16:09:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:09:07.314 INFO:teuthology.orchestra.run.trial020.stderr:16:09:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf 2026-02-24T16:09:07.473 INFO:teuthology.orchestra.run.trial020.stderr:16:09:07 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:09:07.473 INFO:teuthology.orchestra.run.trial020.stderr:16:09:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:10:07.536 INFO:teuthology.orchestra.run.trial020.stderr:16:10:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:10:07.697 INFO:teuthology.orchestra.run.trial020.stderr:16:10:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:10:07.872 INFO:teuthology.orchestra.run.trial020.stderr:16:10:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* 2026-02-24T16:10:08.040 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial020.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-24T16:10:08.197 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - INFO - cbt - Creating fio images... 2026-02-24T16:10:08.198 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl 2026-02-24T16:10:08.354 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:10:08.356 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf 2026-02-24T16:10:08.512 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:10:08.513 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:10:08.514 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2026-02-24T16:10:08.860 INFO:teuthology.orchestra.run.trial020.stderr:16:10:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2026-02-24T16:10:09.839 INFO:teuthology.orchestra.run.trial020.stderr:16:10:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2026-02-24T16:10:10.842 INFO:teuthology.orchestra.run.trial020.stderr:16:10:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2026-02-24T16:10:11.847 INFO:teuthology.orchestra.run.trial020.stderr:16:10:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2026-02-24T16:10:12.850 INFO:teuthology.orchestra.run.trial020.stderr:16:10:12 - INFO - cbt - Checking Health after pool creation. 2026-02-24T16:10:17.856 INFO:teuthology.orchestra.run.trial020.stderr:16:10:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:10:18.215 INFO:teuthology.orchestra.run.trial020.stderr:16:10:18 - INFO - cbt - Final Pool Health Check. 2026-02-24T16:10:23.221 INFO:teuthology.orchestra.run.trial020.stderr:16:10:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:10:23.578 INFO:teuthology.orchestra.run.trial020.stderr:16:10:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com hostname -f 2026-02-24T16:10:23.734 INFO:teuthology.orchestra.run.trial020.stderr:16:10:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial020-0 --size 4096 --pool cbt-librbdfio --order 22 2026-02-24T16:10:25.987 INFO:teuthology.orchestra.run.trial020.stderr:16:10:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial020-1 --size 4096 --pool cbt-librbdfio --order 22 2026-02-24T16:10:26.238 INFO:teuthology.orchestra.run.trial020.stderr:16:10:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:10:26.409 INFO:teuthology.orchestra.run.trial020.stderr:16:10:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:10:26.580 INFO:teuthology.orchestra.run.trial020.stderr:16:10:26 - INFO - cbt - Attempting to prefill fio images... 2026-02-24T16:10:26.580 INFO:teuthology.orchestra.run.trial020.stderr:16:10:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:10:26.581 INFO:teuthology.orchestra.run.trial020.stderr:16:10:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:11:09.909 INFO:teuthology.orchestra.run.trial020.stderr:16:11:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:11:09.911 INFO:teuthology.orchestra.run.trial020.stderr:16:11:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:11:32.746 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-24T16:11:32.747 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be. 2026-02-24T16:11:32.747 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - INFO - cbt - Running benchmark /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-24T16:11:32.748 INFO:teuthology.orchestra.run.trial020.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-24T16:11:32.748 INFO:teuthology.orchestra.run.trial020.stderr:{'cmd_path': '/home/ubuntu/cephtest/fio/fio', 'log_avg_msec': 100, 'norandommap': True, 'pool_profile': 'rbd', 'time': 60, 'vol_size': 4096, 'iodepth': 32, 'mode': 'randrw', 'op_size': 4096, 'osd_ra': 4096, 'procs_per_volume': 1, 'volumes_per_client': 2, 'benchmark': 'librbdfio', 'iteration': 0} == iteration 0 == 2026-02-24T16:11:32.748 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - INFO - cbt - Setting OSD Read Ahead to: 4096 2026-02-24T16:11:32.748 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-24T16:11:32.748 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-24T16:11:32.749 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-24T16:11:32.914 INFO:teuthology.orchestra.run.trial020.stderr:16:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sync 2026-02-24T16:11:33.160 INFO:teuthology.orchestra.run.trial020.stderr:16:11:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-24T16:11:34.341 INFO:teuthology.orchestra.run.trial020.stderr:16:11:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-24T16:11:34.532 INFO:teuthology.orchestra.run.trial020.stderr:16:11:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/ceph_settings.out 2026-02-24T16:11:59.828 INFO:teuthology.orchestra.run.trial020.stderr:16:11:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log 2026-02-24T16:12:00.198 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/collectl 2026-02-24T16:12:00.358 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:12:00.360 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf 2026-02-24T16:12:00.520 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:12:00.520 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.${pid} & done 2026-02-24T16:12:00.522 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - INFO - cbt - Running rbd fio randrw test. 2026-02-24T16:12:00.522 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2026-02-24T16:12:00.522 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 2026-02-24T16:12:00.523 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2026-02-24T16:12:00.523 INFO:teuthology.orchestra.run.trial020.stderr:16:12:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 2026-02-24T16:13:01.266 INFO:teuthology.orchestra.run.trial020.stderr:16:13:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:13:01.447 INFO:teuthology.orchestra.run.trial020.stderr:16:13:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:13:01.634 INFO:teuthology.orchestra.run.trial020.stderr:16:13:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf.data 2026-02-24T16:13:01.636 INFO:teuthology.orchestra.run.trial020.stderr:16:13:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.* 2026-02-24T16:13:01.638 INFO:teuthology.orchestra.run.trial020.stderr:16:13:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/historic_ops.out 2026-02-24T16:13:01.999 INFO:teuthology.orchestra.run.trial020.stderr:16:13:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial020.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* 2026-02-24T16:13:02.167 INFO:teuthology.orchestra.run.trial020.stderr:16:13:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial020.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be 2026-02-24T16:13:02.331 INFO:teuthology.orchestra.run.trial020.stderr:16:13:02 - INFO - cbt - Convert results to json format. 2026-02-24T16:13:02.332 INFO:teuthology.orchestra.run.trial020.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2026-02-24T16:13:02.332 INFO:teuthology.orchestra.run.trial020.stdout:trial020: trial020 2026-02-24T16:13:02.332 INFO:teuthology.orchestra.run.trial020.stdout: 2026-02-24T16:13:02.332 INFO:teuthology.orchestra.run.trial020.stdout:['trial020'] 2026-02-24T16:13:02.345 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-24T16:13:02.360 DEBUG:teuthology.orchestra.run.trial020:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-24T16:13:02.386 DEBUG:teuthology.orchestra.run.trial020:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2026-02-24T16:13:02.464 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-24T16:13:02.465 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-24T16:13:02.465 DEBUG:teuthology.orchestra.run.trial020:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-24T16:13:02.522 INFO:teuthology.orchestra.run.trial020.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.1.trial020.front.sepia.ceph.com 2026-02-24T16:13:02.522 INFO:teuthology.orchestra.run.trial020.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.0.trial020.front.sepia.ceph.com 2026-02-24T16:13:02.523 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:13:02.523 DEBUG:teuthology.orchestra.run.trial020:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.1.trial020.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:13:02.582 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:13:02.582 DEBUG:teuthology.orchestra.run.trial020:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be -name 'perf_stat.*' 2026-02-24T16:13:02.638 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:13:02.638 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:13:02.639 DEBUG:teuthology.orchestra.run.trial020:> set -ex 2026-02-24T16:13:02.639 DEBUG:teuthology.orchestra.run.trial020:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.0.trial020.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:13:02.697 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:13:02.697 DEBUG:teuthology.orchestra.run.trial020:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be -name 'perf_stat.*' 2026-02-24T16:13:02.754 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:13:02.754 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:13:02.759 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-24T16:13:02.760 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-24T16:13:02.760 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-24T16:13:02.775 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-24T16:13:02.775 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-24T16:13:02.776 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial020.front.sepia.ceph.com 2026-02-24T16:13:02.776 DEBUG:teuthology.orchestra.run.trial020:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:13:02.810 DEBUG:teuthology.orchestra.run.trial020:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:13:02.866 DEBUG:teuthology.orchestra.run.trial020:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:13:02.922 INFO:teuthology.orchestra.run.trial020.stdout:"600" 2026-02-24T16:13:02.923 DEBUG:teuthology.orchestra.run.trial020:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:08:41:018193 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:13:02.979 DEBUG:teuthology.orchestra.run.trial020:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:13:03.036 DEBUG:teuthology.orchestra.run.trial020:> rm /home/ubuntu/.ssh/config 2026-02-24T16:13:03.092 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-24T16:13:03.108 DEBUG:teuthology.orchestra.run.trial020:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-24T16:13:03.207 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:13:03.201+0000 7fbe0c3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:13:03.208 INFO:teuthology.orchestra.run.trial020.stderr:2026-02-24T16:13:03.206+0000 7fbe0c3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:13:03.333 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-24T16:13:03.334 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-24T16:13:03.334 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-24T16:13:03.334 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:03.428 INFO:tasks.ceph.osd.0:Stopped 2026-02-24T16:13:03.428 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-24T16:13:03.428 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:08.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-24T16:13:09.530 INFO:tasks.ceph.osd.1:Stopped 2026-02-24T16:13:09.531 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-24T16:13:09.531 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:09.628 INFO:tasks.ceph.osd.2:Stopped 2026-02-24T16:13:09.628 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-24T16:13:09.628 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-24T16:13:09.628 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:09.658 INFO:tasks.ceph.mgr.x:Stopped 2026-02-24T16:13:09.658 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-24T16:13:09.658 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-24T16:13:09.658 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:09.708 INFO:tasks.ceph.mon.a:Stopped 2026-02-24T16:13:09.708 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-24T16:13:09.708 DEBUG:teuthology.orchestra.run.trial020:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-24T16:13:09.773 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:13:09.774 DEBUG:teuthology.orchestra.run.trial020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-24T16:13:09.851 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:13:09.851 DEBUG:teuthology.orchestra.run.trial020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-24T16:13:09.924 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:13:09.924 DEBUG:teuthology.orchestra.run.trial020:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-24T16:13:09.999 INFO:tasks.ceph:Archiving mon data... 2026-02-24T16:13:10.001 DEBUG:teuthology.misc:Transferring archived files from trial020:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68172/data/mon.a.tgz 2026-02-24T16:13:10.001 DEBUG:teuthology.orchestra.run.trial020:> mktemp 2026-02-24T16:13:10.056 INFO:teuthology.orchestra.run.trial020.stdout:/tmp/tmp.QML9QIUHrP 2026-02-24T16:13:10.057 DEBUG:teuthology.orchestra.run.trial020:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QML9QIUHrP 2026-02-24T16:13:10.146 DEBUG:teuthology.orchestra.run.trial020:> sudo chmod 0666 /tmp/tmp.QML9QIUHrP 2026-02-24T16:13:10.228 DEBUG:teuthology.orchestra.remote:trial020:/tmp/tmp.QML9QIUHrP is 207KB 2026-02-24T16:13:10.322 DEBUG:teuthology.orchestra.run.trial020:> rm -fr /tmp/tmp.QML9QIUHrP 2026-02-24T16:13:10.340 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-24T16:13:10.341 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-24T16:13:10.460 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-24T16:13:10.461 INFO:tasks.ceph:Archiving crash dumps... 2026-02-24T16:13:10.463 DEBUG:teuthology.misc:Transferring archived files from trial020:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68172/remote/trial020/crash 2026-02-24T16:13:10.465 DEBUG:teuthology.orchestra.run.trial020:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-24T16:13:10.525 INFO:tasks.ceph:Compressing logs... 2026-02-24T16:13:10.525 DEBUG:teuthology.orchestra.run.trial020:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:13:10.606 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2026-02-24T16:13:10.606 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-24T16:13:10.606 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-24T16:13:10.606 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40324.log: /var/log/ceph/ceph-osd.2.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2026-02-24T16:13:10.606 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2026-02-24T16:13:10.606 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-24T16:13:10.606 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40947.log 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40277.log 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46888.log 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40277.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40947.log.gz 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40996.log 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40093.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40277.log.gz 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.46888.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz -5 2026-02-24T16:13:10.607 INFO:teuthology.orchestra.run.trial020.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46888.log.gz 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40230.log 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40996.log.gz 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.47469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48705.log 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40230.log.gz 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.48705.log: gzip 50.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.48705.log.gz --verbose 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr: -- /var/log/ceph/ceph-client.admin.47289.log 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47287.log 2026-02-24T16:13:10.608 INFO:teuthology.orchestra.run.trial020.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.47289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48038.log 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47289.log.gz 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.47287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39951.log 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47287.log.gz 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.48038.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48038.log.gz 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38655.log 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.39951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39951.log.gz 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph.tmp-client.admin.38655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46923.log 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38655.log.gz 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2026-02-24T16:13:10.609 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.46923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41437.log 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46923.log.gz 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48037.log 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46830.log 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41437.log.gz 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.48037.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48037.log.gz 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41045.log 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.46830.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46830.log.gz 2026-02-24T16:13:10.610 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47468.log 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41045.log.gz 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47802.log 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.47468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40882.log 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr: 86.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47468.log.gz 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.47802.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47802.log.gz 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41196.log 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.40882.log.gz 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41196.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-24T16:13:10.611 INFO:teuthology.orchestra.run.trial020.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41196.log.gz 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.47063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-osd.1.log: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41486.log 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41486.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46771.log 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41486.log.gz 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46656.log 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-24T16:13:10.612 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.46771.log: /var/log/ceph/ceph-client.admin.46656.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46771.log.gz 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46656.log.gz 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.46977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41535.log 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41094.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gzgzip 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41344.log 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40787.log 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41344.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41535.log.gz 2026-02-24T16:13:10.613 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47111.log 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41344.log.gz 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40787.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.47111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40046.log 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40787.log.gz 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.47111.log.gz 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.40046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46712.log 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40046.log.gz 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2026-02-24T16:13:10.614 INFO:teuthology.orchestra.run.trial020.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2026-02-24T16:13:10.615 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.46712.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46712.log.gz 2026-02-24T16:13:10.615 INFO:teuthology.orchestra.run.trial020.stderr:/var/log/ceph/ceph-client.admin.41195.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2026-02-24T16:13:10.615 INFO:teuthology.orchestra.run.trial020.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-24T16:13:13.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-24T16:13:18.634 INFO:teuthology.orchestra.run.trial020.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-24T16:13:18.823 INFO:teuthology.orchestra.run.trial020.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-24T16:13:19.391 INFO:teuthology.orchestra.run.trial020.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-24T16:13:19.393 INFO:teuthology.orchestra.run.trial020.stderr: 2026-02-24T16:13:19.393 INFO:teuthology.orchestra.run.trial020.stderr:real 0m8.811s 2026-02-24T16:13:19.393 INFO:teuthology.orchestra.run.trial020.stderr:user 0m23.702s 2026-02-24T16:13:19.393 INFO:teuthology.orchestra.run.trial020.stderr:sys 0m1.357s 2026-02-24T16:13:19.394 INFO:tasks.ceph:Archiving logs... 2026-02-24T16:13:19.395 DEBUG:teuthology.misc:Transferring archived files from trial020:/var/log/ceph to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68172/remote/trial020/log 2026-02-24T16:13:19.396 DEBUG:teuthology.orchestra.run.trial020:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:13:21.466 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:13:21.541 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:13:21.551 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-24T16:13:21.552 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:13:21.577 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-24T16:13:21.589 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-24T16:13:21.589 DEBUG:teuthology.orchestra.run.trial020:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:13:21.632 INFO:teuthology.orchestra.run.trial020.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:13:21.656 INFO:teuthology.orchestra.run.trial020.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:13:21.656 INFO:teuthology.orchestra.run.trial020.stdout:=============================================================================== 2026-02-24T16:13:21.656 INFO:teuthology.orchestra.run.trial020.stdout:^- connected.by.freedominte> 2 8 377 152 -651us[ -696us] +/- 41ms 2026-02-24T16:13:21.656 INFO:teuthology.orchestra.run.trial020.stdout:^* 159.203.82.102 4 7 377 23 -221us[ -271us] +/- 7714us 2026-02-24T16:13:21.656 INFO:teuthology.orchestra.run.trial020.stdout:^- 172-232-15-202.ip.linode> 2 6 377 26 -7237us[-7287us] +/- 76ms 2026-02-24T16:13:21.656 INFO:teuthology.orchestra.run.trial020.stdout:^- 23.157.160.168 2 7 377 24 +1233us[+1183us] +/- 30ms 2026-02-24T16:13:21.656 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-24T16:13:21.667 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-24T16:13:21.668 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-24T16:13:21.682 DEBUG:teuthology.orchestra.run.trial020:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:13:21.754 DEBUG:teuthology.orchestra.run.trial020:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:13:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:13:21.781 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-24T16:13:21.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-24T16:13:21.812 INFO:teuthology.task.internal:Duration was 2204.470252 seconds 2026-02-24T16:13:21.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-24T16:13:21.828 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-24T16:13:21.829 DEBUG:teuthology.orchestra.run.trial020:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:13:21.868 INFO:teuthology.orchestra.run.trial020.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:13:22.218 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-24T16:13:22.218 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial020.front.sepia.ceph.com 2026-02-24T16:13:22.219 DEBUG:teuthology.orchestra.run.trial020:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:13:22.275 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-24T16:13:22.276 DEBUG:teuthology.orchestra.run.trial020:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:13:22.381 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-24T16:13:22.381 DEBUG:teuthology.orchestra.run.trial020:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:13:22.403 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:13:22.403 INFO:teuthology.orchestra.run.trial020.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:13:22.404 INFO:teuthology.orchestra.run.trial020.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:13:22.404 INFO:teuthology.orchestra.run.trial020.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:13:22.404 INFO:teuthology.orchestra.run.trial020.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:13:22.414 INFO:teuthology.orchestra.run.trial020.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:13:22.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-24T16:13:22.427 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-24T16:13:22.427 DEBUG:teuthology.orchestra.run.trial020:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:13:22.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-24T16:13:22.495 DEBUG:teuthology.orchestra.run.trial020:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:13:22.547 INFO:teuthology.orchestra.run.trial020.stdout:kernel.core_pattern = core 2026-02-24T16:13:22.560 DEBUG:teuthology.orchestra.run.trial020:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:13:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:13:22.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-24T16:13:22.632 INFO:teuthology.task.internal:Transferring archived files... 2026-02-24T16:13:22.633 DEBUG:teuthology.misc:Transferring archived files from trial020:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68172/remote/trial020 2026-02-24T16:13:22.633 DEBUG:teuthology.orchestra.run.trial020:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:13:22.724 INFO:teuthology.task.internal:Removing archive directory... 2026-02-24T16:13:22.724 DEBUG:teuthology.orchestra.run.trial020:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:13:22.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-24T16:13:22.769 INFO:teuthology.task.internal:Not uploading archives. 2026-02-24T16:13:22.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-24T16:13:22.786 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-24T16:13:22.786 DEBUG:teuthology.orchestra.run.trial020:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:13:22.804 INFO:teuthology.orchestra.run.trial020.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:13 /home/ubuntu/cephtest 2026-02-24T16:13:22.806 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-24T16:13:22.816 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-24T16:13:22.855 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/fio_4K_rand_rw} duration: 2204.4702517986298 flavor: debug owner: scheduled_liucm@soko04.front.sepia.ceph.com success: true 2026-02-24T16:13:22.855 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:13:22.960 INFO:teuthology.run:pass