2026-02-24T15:34:08.328 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T15:34:08.344 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:34:08.422 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68175 branch: wip-liucm-shard-redesign-crimson-only description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/fio_4M_rand_read} email: null first_in_suite: false flavor: debug job_id: '68175' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-liucm-shard-redesign-crimson-only ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 workunit: branch: wip-liucm-shard-redesign-crimson-only env: CRIMSON_COMPAT: '1' sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 owner: scheduled_liucm@soko04.front.sepia.ceph.com priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9834 sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-liucm-shard-redesign-crimson-only suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 targets: trial195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNs2UmTz9GKjdOR/PSlituT9KkcsRPvZIq7MOfrVjnXdYa45Vw1aHHseeIGSVGD+RaG7gEjWuf1pguoNch9sx48= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-24_15:28:57 tube: trial user: liucm verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-24T15:34:08.422 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa; will attempt to use it 2026-02-24T15:34:08.423 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa/tasks 2026-02-24T15:34:08.423 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-24T15:34:08.424 INFO:teuthology.task.internal:Checking packages... 2026-02-24T15:34:08.424 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1' 2026-02-24T15:34:08.424 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-24T15:34:08.425 INFO:teuthology.packaging:ref: None 2026-02-24T15:34:08.425 INFO:teuthology.packaging:tag: None 2026-02-24T15:34:08.425 INFO:teuthology.packaging:branch: wip-liucm-shard-redesign-crimson-only 2026-02-24T15:34:08.425 INFO:teuthology.packaging:sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:34:08.425 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-liucm-shard-redesign-crimson-only 2026-02-24T15:34:08.549 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5343.gd26e5c22 2026-02-24T15:34:08.550 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-24T15:34:08.561 INFO:teuthology.task.internal:no buildpackages task found 2026-02-24T15:34:08.562 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-24T15:34:08.574 INFO:teuthology.task.internal:Saving configuration 2026-02-24T15:34:08.588 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-24T15:34:08.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-24T15:34:08.664 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial195.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68175', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 15:30:37.271927', 'locked_by': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNs2UmTz9GKjdOR/PSlituT9KkcsRPvZIq7MOfrVjnXdYa45Vw1aHHseeIGSVGD+RaG7gEjWuf1pguoNch9sx48='} 2026-02-24T15:34:08.664 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-24T15:34:08.678 INFO:teuthology.task.internal:roles: ubuntu@trial195.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-24T15:34:08.678 INFO:teuthology.run_tasks:Running task console_log... 2026-02-24T15:34:08.757 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x71609e282980>, signals=[15]) 2026-02-24T15:34:08.757 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-24T15:34:08.768 INFO:teuthology.task.internal:Opening connections... 2026-02-24T15:34:08.768 DEBUG:teuthology.task.internal:connecting to ubuntu@trial195.front.sepia.ceph.com 2026-02-24T15:34:08.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:34:08.832 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-24T15:34:08.842 DEBUG:teuthology.orchestra.run.trial195:> uname -m 2026-02-24T15:34:08.906 INFO:teuthology.orchestra.run.trial195.stdout:x86_64 2026-02-24T15:34:08.906 DEBUG:teuthology.orchestra.run.trial195:> cat /etc/os-release 2026-02-24T15:34:08.962 INFO:teuthology.orchestra.run.trial195.stdout:NAME="CentOS Stream" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:VERSION="9" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:ID="centos" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:ID_LIKE="rhel fedora" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:VERSION_ID="9" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:PLATFORM_ID="platform:el9" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:ANSI_COLOR="0;31" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:LOGO="fedora-logo-icon" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:HOME_URL="https://centos.org/" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T15:34:08.963 INFO:teuthology.orchestra.run.trial195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T15:34:08.964 INFO:teuthology.lock.ops:Updating trial195.front.sepia.ceph.com on lock server 2026-02-24T15:34:09.050 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-24T15:34:09.062 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-24T15:34:09.074 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-24T15:34:09.074 DEBUG:teuthology.orchestra.run.trial195:> test '!' -e /home/ubuntu/cephtest 2026-02-24T15:34:09.089 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-24T15:34:09.100 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-24T15:34:09.100 DEBUG:teuthology.orchestra.run.trial195:> test -z $(ls -A /var/lib/ceph) 2026-02-24T15:34:09.146 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-24T15:34:09.251 INFO:teuthology.run_tasks:Running task kernel... 2026-02-24T15:34:09.272 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-24T15:34:09.272 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-24T15:34:09.272 DEBUG:teuthology.orchestra.run.trial195:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T15:34:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:34:09.287 DEBUG:teuthology.orchestra.run.trial195:> uname -r 2026-02-24T15:34:09.343 INFO:teuthology.orchestra.run.trial195.stdout:5.14.0-681.el9.x86_64 2026-02-24T15:34:09.344 INFO:teuthology.task.kernel:Running kernel on trial195: 5.14.0-681.el9.x86_64 2026-02-24T15:34:09.344 DEBUG:teuthology.orchestra.run.trial195:> sudo yum install -y kernel 2026-02-24T15:34:10.762 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.9 MB 00:01 2026-02-24T15:34:14.140 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 27 MB 00:02 2026-02-24T15:34:18.119 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 8.0 MB 00:01 2026-02-24T15:34:19.343 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 20 kB 00:00 2026-02-24T15:34:20.052 INFO:teuthology.orchestra.run.trial195.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2026-02-24T15:34:23.254 INFO:teuthology.orchestra.run.trial195.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-24T15:34:24.269 INFO:teuthology.orchestra.run.trial195.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T15:34:24.270 INFO:teuthology.orchestra.run.trial195.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T15:34:24.289 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:34:24.291 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:34:24.291 INFO:teuthology.orchestra.run.trial195.stdout: Package Architecture Version Repository Size 2026-02-24T15:34:24.291 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:34:24.291 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:34:24.291 INFO:teuthology.orchestra.run.trial195.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T15:34:24.291 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:34:24.291 INFO:teuthology.orchestra.run.trial195.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout:Install 4 Packages 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 89 M 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 135 M 2026-02-24T15:34:24.292 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:34:24.827 INFO:teuthology.orchestra.run.trial195.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 4.1 MB/s | 901 kB 00:00 2026-02-24T15:34:25.032 INFO:teuthology.orchestra.run.trial195.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 41 MB/s | 17 MB 00:00 2026-02-24T15:34:25.223 INFO:teuthology.orchestra.run.trial195.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 65 MB/s | 40 MB 00:00 2026-02-24T15:34:25.267 INFO:teuthology.orchestra.run.trial195.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 71 MB/s | 31 MB 00:00 2026-02-24T15:34:25.267 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:34:25.267 INFO:teuthology.orchestra.run.trial195.stdout:Total 91 MB/s | 89 MB 00:00 2026-02-24T15:34:25.460 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:34:25.663 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:34:25.663 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:34:26.048 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:34:26.048 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:34:27.158 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:34:27.446 INFO:teuthology.orchestra.run.trial195.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:34:27.468 INFO:teuthology.orchestra.run.trial195.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:34:30.040 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:34:30.073 INFO:teuthology.orchestra.run.trial195.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:34:33.176 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:34:33.182 INFO:teuthology.orchestra.run.trial195.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:36.229 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:53.690 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:53.712 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:53.898 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:53.898 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:34:53.898 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:34:53.898 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:34:53.955 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:34:53.955 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:34:53.955 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:34:53.955 INFO:teuthology.orchestra.run.trial195.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T15:34:53.955 INFO:teuthology.orchestra.run.trial195.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T15:34:53.955 INFO:teuthology.orchestra.run.trial195.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T15:34:53.956 INFO:teuthology.orchestra.run.trial195.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T15:34:53.956 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:34:53.956 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:34:54.018 DEBUG:teuthology.orchestra.run.trial195:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:34:54.047 INFO:teuthology.orchestra.run.trial195.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.048 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.049 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.049 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T15:34:54.049 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T15:34:54.049 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-24T15:34:54.049 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial195.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T15:34:54.049 DEBUG:teuthology.orchestra.run.trial195:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:34:54.113 INFO:teuthology.orchestra.run.trial195.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.115 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.115 DEBUG:teuthology.orchestra.run.trial195:> sudo rpm -qi grub2-tools 2026-02-24T15:34:54.183 INFO:teuthology.orchestra.run.trial195.stdout:Name : grub2-tools 2026-02-24T15:34:54.183 INFO:teuthology.orchestra.run.trial195.stdout:Epoch : 1 2026-02-24T15:34:54.183 INFO:teuthology.orchestra.run.trial195.stdout:Version : 2.06 2026-02-24T15:34:54.183 INFO:teuthology.orchestra.run.trial195.stdout:Release : 120.el9 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Architecture: x86_64 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Group : Unspecified 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Size : 9108611 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:License : GPLv3+ 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Packager : builder@centos.org 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:Vendor : CentOS 2026-02-24T15:34:54.184 INFO:teuthology.orchestra.run.trial195.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout:Summary : Support tools for GRUB. 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout:Description : 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout:hardware devices. 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:34:54.185 INFO:teuthology.orchestra.run.trial195.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T15:34:54.186 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T15:34:54.186 INFO:teuthology.task.kernel:Updating grub on trial195 to boot 5.14.0-686.el9.x86_64 2026-02-24T15:34:54.187 DEBUG:teuthology.orchestra.run.trial195:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T15:34:54.249 INFO:teuthology.orchestra.run.trial195.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T15:34:54.251 DEBUG:teuthology.orchestra.run.trial195:> sudo command -v grubby && echo yes || echo no 2026-02-24T15:34:54.314 INFO:teuthology.orchestra.run.trial195.stdout:/sbin/grubby 2026-02-24T15:34:54.315 INFO:teuthology.orchestra.run.trial195.stdout:yes 2026-02-24T15:34:54.315 DEBUG:teuthology.orchestra.run.trial195:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T15:34:54.379 INFO:teuthology.orchestra.run.trial195.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T15:34:54.381 DEBUG:teuthology.orchestra.run.trial195:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.454 DEBUG:teuthology.orchestra.run.trial195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T15:34:54.655 INFO:teuthology.orchestra.run.trial195.stderr:Generating grub configuration file ... 2026-02-24T15:34:54.722 INFO:teuthology.orchestra.run.trial195.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T15:34:54.728 INFO:teuthology.orchestra.run.trial195.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T15:34:54.979 INFO:teuthology.orchestra.run.trial195.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T15:34:54.986 INFO:teuthology.orchestra.run.trial195.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T15:34:55.019 INFO:teuthology.orchestra.run.trial195.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T15:34:55.026 INFO:teuthology.orchestra.run.trial195.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T15:34:55.057 INFO:teuthology.orchestra.run.trial195.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T15:34:55.063 INFO:teuthology.orchestra.run.trial195.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T15:34:55.087 INFO:teuthology.orchestra.run.trial195.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T15:34:55.093 INFO:teuthology.orchestra.run.trial195.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T15:34:55.178 INFO:teuthology.orchestra.run.trial195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T15:34:55.187 INFO:teuthology.orchestra.run.trial195.stderr:done 2026-02-24T15:34:55.189 DEBUG:teuthology.orchestra.run.trial195:> test -d /sys/firmware/efi 2026-02-24T15:34:55.205 DEBUG:teuthology.orchestra.run.trial195:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T15:34:55.272 INFO:teuthology.orchestra.run.trial195.stdout:/boot/efi/EFI/BOOT 2026-02-24T15:34:55.272 INFO:teuthology.orchestra.run.trial195.stdout:/boot/efi/EFI/centos 2026-02-24T15:34:55.274 DEBUG:teuthology.orchestra.run.trial195:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T15:34:55.338 INFO:teuthology.orchestra.run.trial195.stdout:no 2026-02-24T15:34:55.339 DEBUG:teuthology.orchestra.run.trial195:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T15:34:55.402 INFO:teuthology.orchestra.run.trial195.stdout:yes 2026-02-24T15:34:55.403 DEBUG:teuthology.orchestra.run.trial195:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T15:34:55.468 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial195 2026-02-24T15:34:55.468 DEBUG:teuthology.orchestra.run.trial195:> sudo bash -c ' 2026-02-24T15:34:55.469 DEBUG:teuthology.orchestra.run.trial195:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T15:34:55.469 DEBUG:teuthology.orchestra.run.trial195:> ' 2026-02-24T15:35:25.510 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-24T15:35:25.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial195.front.sepia.ceph.com' 2026-02-24T15:35:25.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:36:25.574 DEBUG:teuthology.orchestra.remote:timed out 2026-02-24T15:36:34.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial195.front.sepia.ceph.com' 2026-02-24T15:36:34.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:36:34.722 DEBUG:teuthology.orchestra.run.trial195:> true 2026-02-24T15:36:34.878 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial195.front.sepia.ceph.com' 2026-02-24T15:36:34.878 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-24T15:36:34.878 DEBUG:teuthology.orchestra.run.trial195:> uname -r 2026-02-24T15:36:34.934 INFO:teuthology.orchestra.run.trial195.stdout:5.14.0-686.el9.x86_64 2026-02-24T15:36:34.934 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T15:36:34.935 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T15:36:34.935 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T15:36:35.936 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-24T15:36:35.936 DEBUG:teuthology.orchestra.run.trial195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T15:36:35.966 INFO:teuthology.orchestra.run.trial195.stdout:ttyS1 2026-02-24T15:36:35.992 DEBUG:teuthology.parallel:result is None 2026-02-24T15:36:35.993 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-24T15:36:36.004 INFO:teuthology.task.internal:Creating test directory... 2026-02-24T15:36:36.004 DEBUG:teuthology.orchestra.run.trial195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T15:36:36.049 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-24T15:36:36.059 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-24T15:36:36.072 INFO:teuthology.task.internal:Creating archive directory... 2026-02-24T15:36:36.072 DEBUG:teuthology.orchestra.run.trial195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T15:36:36.112 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-24T15:36:36.123 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-24T15:36:36.124 DEBUG:teuthology.orchestra.run.trial195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T15:36:36.170 INFO:teuthology.orchestra.run.trial195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:36:36.179 INFO:teuthology.orchestra.run.trial195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:36:36.181 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-24T15:36:36.193 INFO:teuthology.task.internal:Configuring sudo... 2026-02-24T15:36:36.193 DEBUG:teuthology.orchestra.run.trial195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T15:36:36.247 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-24T15:36:36.266 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-24T15:36:36.267 DEBUG:teuthology.orchestra.run.trial195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T15:36:36.302 DEBUG:teuthology.orchestra.run.trial195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:36:36.463 DEBUG:teuthology.orchestra.run.trial195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:36:36.488 DEBUG:teuthology.orchestra.run.trial195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:36:36.634 DEBUG:teuthology.orchestra.run.trial195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:36:36.659 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T15:36:36.659 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T15:36:36.723 DEBUG:teuthology.orchestra.run.trial195:> sudo service rsyslog restart 2026-02-24T15:36:36.791 INFO:teuthology.orchestra.run.trial195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T15:36:37.188 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-24T15:36:37.199 INFO:teuthology.task.internal:Starting timer... 2026-02-24T15:36:37.199 INFO:teuthology.run_tasks:Running task pcp... 2026-02-24T15:36:37.220 INFO:teuthology.run_tasks:Running task selinux... 2026-02-24T15:36:37.238 DEBUG:teuthology.orchestra.run.trial195:> sudo service auditd rotate 2026-02-24T15:36:37.266 INFO:teuthology.orchestra.run.trial195.stdout:Rotating logs: 2026-02-24T15:36:37.268 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-24T15:36:37.268 DEBUG:teuthology.orchestra.run.trial195:> /usr/sbin/getenforce 2026-02-24T15:36:37.325 INFO:teuthology.orchestra.run.trial195.stdout:Permissive 2026-02-24T15:36:37.326 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial195.front.sepia.ceph.com': 'permissive'} 2026-02-24T15:36:37.326 DEBUG:teuthology.orchestra.run.trial195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T15:36:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:36:37.391 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-24T15:36:37.392 DEBUG:teuthology.orchestra.run.trial195:> sudo /usr/sbin/setenforce permissive 2026-02-24T15:36:37.456 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-24T15:36:37.471 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-24T15:36:37.485 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-24T15:36:37.486 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-24T15:39:19.647 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial195.front.sepia.ceph.com')] 2026-02-24T15:39:19.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial195.front.sepia.ceph.com' 2026-02-24T15:39:19.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:39:19.717 DEBUG:teuthology.orchestra.run.trial195:> true 2026-02-24T15:39:19.794 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial195.front.sepia.ceph.com' 2026-02-24T15:39:19.794 INFO:teuthology.run_tasks:Running task clock... 2026-02-24T15:39:19.808 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-24T15:39:19.808 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T15:39:19.808 DEBUG:teuthology.orchestra.run.trial195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T15:39:19.861 INFO:teuthology.orchestra.run.trial195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T15:39:19.873 INFO:teuthology.orchestra.run.trial195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T15:39:19.894 INFO:teuthology.orchestra.run.trial195.stderr:sudo: ntpd: command not found 2026-02-24T15:39:19.904 INFO:teuthology.orchestra.run.trial195.stdout:506 Cannot talk to daemon 2026-02-24T15:39:19.915 INFO:teuthology.orchestra.run.trial195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T15:39:19.928 INFO:teuthology.orchestra.run.trial195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T15:39:19.974 INFO:teuthology.orchestra.run.trial195.stderr:bash: line 1: ntpq: command not found 2026-02-24T15:39:19.992 INFO:teuthology.orchestra.run.trial195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T15:39:19.992 INFO:teuthology.orchestra.run.trial195.stdout:=============================================================================== 2026-02-24T15:39:19.992 INFO:teuthology.orchestra.run.trial195.stdout:^? ntp.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:39:19.992 INFO:teuthology.orchestra.run.trial195.stdout:^? ntp.silvertree.org 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:39:19.992 INFO:teuthology.orchestra.run.trial195.stdout:^? ip212-227-240-160.pbiaas> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:39:19.992 INFO:teuthology.orchestra.run.trial195.stdout:^? 23.186.168.132 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:39:19.993 INFO:teuthology.run_tasks:Running task install... 2026-02-24T15:39:20.009 DEBUG:teuthology.task.install:project ceph 2026-02-24T15:39:20.009 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T15:39:20.009 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T15:39:20.009 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T15:39:20.019 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T15:39:20.019 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-24T15:39:20.020 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T15:39:20.021 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:39:20.184 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T15:39:20.184 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T15:39:20.312 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T15:39:20.313 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T15:39:20.313 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T15:39:20.338 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-24T15:39:20.338 DEBUG:teuthology.orchestra.run.trial195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T15:39:20.403 DEBUG:teuthology.orchestra.run.trial195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T15:39:20.476 DEBUG:teuthology.orchestra.run.trial195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T15:39:20.539 INFO:teuthology.orchestra.run.trial195.stdout:check_obsoletes = 1 2026-02-24T15:39:20.541 DEBUG:teuthology.orchestra.run.trial195:> sudo yum clean all 2026-02-24T15:39:20.688 INFO:teuthology.orchestra.run.trial195.stdout:41 files removed 2026-02-24T15:39:20.702 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-radosgw 2026-02-24T15:39:20.981 INFO:teuthology.orchestra.run.trial195.stdout:ceph packages for x86_64 713 kB/s | 92 kB 00:00 2026-02-24T15:39:21.141 INFO:teuthology.orchestra.run.trial195.stdout:ceph noarch packages 180 kB/s | 24 kB 00:00 2026-02-24T15:39:21.284 INFO:teuthology.orchestra.run.trial195.stdout:ceph source packages 17 kB/s | 2.1 kB 00:00 2026-02-24T15:39:22.111 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-24T15:39:30.472 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - AppStream 3.4 MB/s | 27 MB 00:07 2026-02-24T15:39:33.812 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 8.0 MB 00:00 2026-02-24T15:39:34.752 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 20 kB 00:00 2026-02-24T15:39:35.259 INFO:teuthology.orchestra.run.trial195.stdout:Extra Packages for Enterprise Linux 47 MB/s | 20 MB 00:00 2026-02-24T15:39:38.515 INFO:teuthology.orchestra.run.trial195.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-24T15:39:39.738 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:39:39.739 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: ceph-radosgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 11 M 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout:Upgrading: 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: librados2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 3.3 M 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: librbd1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.7 M 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: ceph-base x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.1 M 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 21 M 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: ceph-selinux x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 25 k 2026-02-24T15:39:39.740 INFO:teuthology.orchestra.run.trial195.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: libcephfs2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 860 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: libradosstriper1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 263 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: librgw2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.6 M 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T15:39:39.741 INFO:teuthology.orchestra.run.trial195.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: python3-ceph-argparse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 45 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: python3-ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 185 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: python3-cephfs x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 203 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: python3-rados x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 324 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: python3-rbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 305 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: python3-rgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 100 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout:Installing weak dependencies: 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:39:39.742 INFO:teuthology.orchestra.run.trial195.stdout:Install 32 Packages 2026-02-24T15:39:39.743 INFO:teuthology.orchestra.run.trial195.stdout:Upgrade 2 Packages 2026-02-24T15:39:39.743 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:39.743 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 60 M 2026-02-24T15:39:39.743 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:39:41.678 INFO:teuthology.orchestra.run.trial195.stdout:(1/34): ceph-base-20.3.0-5343.gd26e5c22.el9.x86 3.0 MB/s | 5.1 MB 00:01 2026-02-24T15:39:41.707 INFO:teuthology.orchestra.run.trial195.stdout:(2/34): ceph-selinux-20.3.0-5343.gd26e5c22.el9. 868 kB/s | 25 kB 00:00 2026-02-24T15:39:41.867 INFO:teuthology.orchestra.run.trial195.stdout:(3/34): libcephfs2-20.3.0-5343.gd26e5c22.el9.x8 5.3 MB/s | 860 kB 00:00 2026-02-24T15:39:41.933 INFO:teuthology.orchestra.run.trial195.stdout:(4/34): libradosstriper1-20.3.0-5343.gd26e5c22. 3.9 MB/s | 263 kB 00:00 2026-02-24T15:39:42.946 INFO:teuthology.orchestra.run.trial195.stdout:(5/34): librgw2-20.3.0-5343.gd26e5c22.el9.x86_6 5.6 MB/s | 5.6 MB 00:01 2026-02-24T15:39:42.967 INFO:teuthology.orchestra.run.trial195.stdout:(6/34): python3-ceph-argparse-20.3.0-5343.gd26e 2.1 MB/s | 45 kB 00:00 2026-02-24T15:39:42.990 INFO:teuthology.orchestra.run.trial195.stdout:(7/34): python3-ceph-common-20.3.0-5343.gd26e5c 8.0 MB/s | 185 kB 00:00 2026-02-24T15:39:43.023 INFO:teuthology.orchestra.run.trial195.stdout:(8/34): python3-cephfs-20.3.0-5343.gd26e5c22.el 6.0 MB/s | 203 kB 00:00 2026-02-24T15:39:43.059 INFO:teuthology.orchestra.run.trial195.stdout:(9/34): python3-rados-20.3.0-5343.gd26e5c22.el9 8.8 MB/s | 324 kB 00:00 2026-02-24T15:39:43.097 INFO:teuthology.orchestra.run.trial195.stdout:(10/34): python3-rbd-20.3.0-5343.gd26e5c22.el9. 8.1 MB/s | 305 kB 00:00 2026-02-24T15:39:43.115 INFO:teuthology.orchestra.run.trial195.stdout:(11/34): python3-rgw-20.3.0-5343.gd26e5c22.el9. 5.3 MB/s | 100 kB 00:00 2026-02-24T15:39:43.377 INFO:teuthology.orchestra.run.trial195.stdout:(12/34): mailcap-2.1.49-5.el9.noarch.rpm 127 kB/s | 33 kB 00:00 2026-02-24T15:39:43.455 INFO:teuthology.orchestra.run.trial195.stdout:(13/34): ceph-radosgw-20.3.0-5343.gd26e5c22.el9 3.2 MB/s | 11 MB 00:03 2026-02-24T15:39:43.604 INFO:teuthology.orchestra.run.trial195.stdout:(14/34): pciutils-3.7.0-7.el9.x86_64.rpm 410 kB/s | 93 kB 00:00 2026-02-24T15:39:43.749 INFO:teuthology.orchestra.run.trial195.stdout:(15/34): ceph-common-20.3.0-5343.gd26e5c22.el9. 5.5 MB/s | 21 MB 00:03 2026-02-24T15:39:43.926 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:44.014 INFO:teuthology.orchestra.run.trial195.stdout:(16/34): boost-program-options-1.75.0-13.el9.x8 186 kB/s | 104 kB 00:00 2026-02-24T15:39:44.073 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:44.098 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:44.268 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:44.361 INFO:teuthology.orchestra.run.trial195.stdout:(17/34): libnbd-1.20.3-4.el9.x86_64.rpm 217 kB/s | 164 kB 00:00 2026-02-24T15:39:44.401 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:44.426 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:44.449 INFO:teuthology.orchestra.run.trial195.stdout:(18/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 229 kB/s | 160 kB 00:00 2026-02-24T15:39:44.475 INFO:teuthology.orchestra.run.trial195.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 11 MB/s | 292 kB 00:00 2026-02-24T15:39:44.489 INFO:teuthology.orchestra.run.trial195.stdout:(20/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 95 kB/s | 45 kB 00:00 2026-02-24T15:39:44.504 INFO:teuthology.orchestra.run.trial195.stdout:(21/34): qatlib-25.08.0-2.el9.x86_64.rpm 8.0 MB/s | 240 kB 00:00 2026-02-24T15:39:44.510 INFO:teuthology.orchestra.run.trial195.stdout:(22/34): qatlib-service-25.08.0-2.el9.x86_64.rp 1.7 MB/s | 37 kB 00:00 2026-02-24T15:39:44.530 INFO:teuthology.orchestra.run.trial195.stdout:(23/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.6 MB/s | 66 kB 00:00 2026-02-24T15:39:44.689 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:39:44.717 INFO:teuthology.orchestra.run.trial195.stdout:(24/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.5 MB/s | 308 kB 00:00 2026-02-24T15:39:44.736 INFO:teuthology.orchestra.run.trial195.stdout:(25/34): librdkafka-1.6.1-102.el9.x86_64.rpm 1.7 MB/s | 662 kB 00:00 2026-02-24T15:39:44.760 INFO:teuthology.orchestra.run.trial195.stdout:(26/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 580 kB/s | 25 kB 00:00 2026-02-24T15:39:44.829 INFO:teuthology.orchestra.run.trial195.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 969 kB/s | 67 kB 00:00 2026-02-24T15:39:44.848 INFO:teuthology.orchestra.run.trial195.stdout:(28/34): libarrow-9.0.0-15.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2026-02-24T15:39:44.876 INFO:teuthology.orchestra.run.trial195.stdout:(29/34): liboath-2.6.12-1.el9.x86_64.rpm 352 kB/s | 49 kB 00:00 2026-02-24T15:39:44.924 INFO:teuthology.orchestra.run.trial195.stdout:(30/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 8.7 MB/s | 838 kB 00:00 2026-02-24T15:39:44.966 INFO:teuthology.orchestra.run.trial195.stdout:(31/34): re2-20211101-20.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2026-02-24T15:39:45.048 INFO:teuthology.orchestra.run.trial195.stdout:(32/34): thrift-0.15.0-4.el9.x86_64.rpm 9.2 MB/s | 1.6 MB 00:00 2026-02-24T15:39:45.560 INFO:teuthology.orchestra.run.trial195.stdout:(33/34): librbd1-20.3.0-5343.gd26e5c22.el9.x86_ 4.5 MB/s | 2.7 MB 00:00 2026-02-24T15:39:45.675 INFO:teuthology.orchestra.run.trial195.stdout:(34/34): librados2-20.3.0-5343.gd26e5c22.el9.x8 4.4 MB/s | 3.3 MB 00:00 2026-02-24T15:39:45.677 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:39:45.677 INFO:teuthology.orchestra.run.trial195.stdout:Total 10 MB/s | 60 MB 00:05 2026-02-24T15:39:45.761 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:39:45.779 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:39:45.779 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:39:46.018 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:39:46.018 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:39:46.298 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:39:46.447 INFO:teuthology.orchestra.run.trial195.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-24T15:39:46.450 INFO:teuthology.orchestra.run.trial195.stdout: Upgrading : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:39:46.482 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:39:46.522 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:39:46.522 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:39:46.534 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:39:46.541 INFO:teuthology.orchestra.run.trial195.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-24T15:39:46.544 INFO:teuthology.orchestra.run.trial195.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-24T15:39:46.547 INFO:teuthology.orchestra.run.trial195.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-24T15:39:46.551 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-24T15:39:46.562 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 9/36 2026-02-24T15:39:46.575 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:39:46.576 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:39:46.596 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:39:46.600 INFO:teuthology.orchestra.run.trial195.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-24T15:39:46.630 INFO:teuthology.orchestra.run.trial195.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-24T15:39:46.634 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-24T15:39:46.640 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-24T15:39:46.642 INFO:teuthology.orchestra.run.trial195.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-24T15:39:46.646 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-24T15:39:46.766 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-24T15:39:46.767 INFO:teuthology.orchestra.run.trial195.stdout: Upgrading : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:39:46.796 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:39:46.800 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 20/36 2026-02-24T15:39:46.805 INFO:teuthology.orchestra.run.trial195.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-24T15:39:47.062 INFO:teuthology.orchestra.run.trial195.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-24T15:39:47.064 INFO:teuthology.orchestra.run.trial195.stdout: Installing : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:39:47.082 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:39:47.086 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 24/36 2026-02-24T15:39:47.090 INFO:teuthology.orchestra.run.trial195.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-24T15:39:47.092 INFO:teuthology.orchestra.run.trial195.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-24T15:39:47.093 INFO:teuthology.orchestra.run.trial195.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:39:47.112 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:39:47.115 INFO:teuthology.orchestra.run.trial195.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-24T15:39:47.148 INFO:teuthology.orchestra.run.trial195.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-24T15:39:47.150 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 30/36 2026-02-24T15:39:48.125 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:39:48.127 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:39:48.394 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:39:48.397 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:39:48.439 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:39:48.439 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T15:39:48.439 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T15:39:48.440 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:48.443 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:39:53.005 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout:skipping the directory /sys 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout:skipping the directory /proc 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout:skipping the directory /mnt 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout:skipping the directory /var/tmp 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout:skipping the directory /home 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout:skipping the directory /root 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout:skipping the directory /tmp 2026-02-24T15:39:53.006 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:53.514 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:39:53.537 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:39:53.537 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:39:53.537 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T15:39:53.537 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:39:53.537 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:39:53.537 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:53.539 INFO:teuthology.orchestra.run.trial195.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:39:53.551 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:39:53.551 INFO:teuthology.orchestra.run.trial195.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:39:53.608 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:39:53.608 INFO:teuthology.orchestra.run.trial195.stdout:Creating group 'qat' with GID 992. 2026-02-24T15:39:53.608 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 7/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 8/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 9/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:39:53.896 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 13/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-24T15:39:53.897 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 35/36 2026-02-24T15:39:53.964 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:39:53.964 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:53.964 INFO:teuthology.orchestra.run.trial195.stdout:Upgraded: 2026-02-24T15:39:53.964 INFO:teuthology.orchestra.run.trial195.stdout: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.964 INFO:teuthology.orchestra.run.trial195.stdout: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.964 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:39:53.964 INFO:teuthology.orchestra.run.trial195.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:53.965 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:39:54.007 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-test 2026-02-24T15:39:54.322 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:16 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:39:54.409 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:39:54.409 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: ceph-test x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 126 M 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: libcephsqlite x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 153 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:Install 12 Packages 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 131 M 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 981 M 2026-02-24T15:39:54.410 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:39:55.096 INFO:teuthology.orchestra.run.trial195.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 8.9 MB/s | 2.1 MB 00:00 2026-02-24T15:39:55.122 INFO:teuthology.orchestra.run.trial195.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 897 kB/s | 23 kB 00:00 2026-02-24T15:39:55.148 INFO:teuthology.orchestra.run.trial195.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 3.5 MB/s | 94 kB 00:00 2026-02-24T15:39:55.352 INFO:teuthology.orchestra.run.trial195.stdout:(4/12): libcephsqlite-20.3.0-5343.gd26e5c22.el9 312 kB/s | 153 kB 00:00 2026-02-24T15:39:56.006 INFO:teuthology.orchestra.run.trial195.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 612 kB/s | 525 kB 00:00 2026-02-24T15:39:56.082 INFO:teuthology.orchestra.run.trial195.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 241 kB/s | 176 kB 00:00 2026-02-24T15:39:56.761 INFO:teuthology.orchestra.run.trial195.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 309 kB/s | 233 kB 00:00 2026-02-24T15:39:57.007 INFO:teuthology.orchestra.run.trial195.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2026-02-24T15:39:57.556 INFO:teuthology.orchestra.run.trial195.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 382 kB/s | 303 kB 00:00 2026-02-24T15:39:57.638 INFO:teuthology.orchestra.run.trial195.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 101 kB/s | 64 kB 00:00 2026-02-24T15:39:57.672 INFO:teuthology.orchestra.run.trial195.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.0 MB/s | 122 kB 00:00 2026-02-24T15:42:44.923 INFO:teuthology.orchestra.run.trial195.stdout:(12/12): ceph-test-20.3.0-5343.gd26e5c22.el9.x8 760 kB/s | 126 MB 02:50 2026-02-24T15:42:44.924 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:42:44.925 INFO:teuthology.orchestra.run.trial195.stdout:Total 787 kB/s | 131 MB 02:50 2026-02-24T15:42:44.996 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:42:45.007 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:42:45.007 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:42:45.436 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:42:45.436 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:42:45.893 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:42:45.902 INFO:teuthology.orchestra.run.trial195.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-24T15:42:45.911 INFO:teuthology.orchestra.run.trial195.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-24T15:42:45.916 INFO:teuthology.orchestra.run.trial195.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-24T15:42:45.920 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-24T15:42:45.922 INFO:teuthology.orchestra.run.trial195.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-24T15:42:45.926 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-24T15:42:45.927 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:42:45.943 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:42:45.945 INFO:teuthology.orchestra.run.trial195.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-24T15:42:45.956 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-24T15:42:45.964 INFO:teuthology.orchestra.run.trial195.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-24T15:42:45.965 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:42:53.112 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:42:53.131 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-24T15:42:53.938 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-24T15:42:53.939 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-24T15:42:53.939 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-24T15:42:53.939 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-24T15:42:53.939 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-24T15:42:53.996 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:42:53.997 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:42:54.038 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph 2026-02-24T15:42:54.357 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:03:16 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:42:54.442 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:42:54.443 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:42:54.444 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:42:54.444 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:42:54.444 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:42:54.444 INFO:teuthology.orchestra.run.trial195.stdout: ceph x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 6.5 k 2026-02-24T15:42:54.444 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:42:54.444 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mds x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.0 M 2026-02-24T15:42:54.444 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 937 k 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 297 k 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mon x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 4.9 M 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 163 k 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd-classic x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 15 M 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd-crimson x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 88 M 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T15:42:54.445 INFO:teuthology.orchestra.run.trial195.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T15:42:54.446 INFO:teuthology.orchestra.run.trial195.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T15:42:54.446 INFO:teuthology.orchestra.run.trial195.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T15:42:54.446 INFO:teuthology.orchestra.run.trial195.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T15:42:54.446 INFO:teuthology.orchestra.run.trial195.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T15:42:54.446 INFO:teuthology.orchestra.run.trial195.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T15:42:54.446 INFO:teuthology.orchestra.run.trial195.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T15:42:54.446 INFO:teuthology.orchestra.run.trial195.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-collections 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T15:42:54.447 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T15:42:54.448 INFO:teuthology.orchestra.run.trial195.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T15:42:54.449 INFO:teuthology.orchestra.run.trial195.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T15:42:54.449 INFO:teuthology.orchestra.run.trial195.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T15:42:54.449 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:42:54.449 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:42:54.449 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:42:54.449 INFO:teuthology.orchestra.run.trial195.stdout:Install 42 Packages 2026-02-24T15:42:54.449 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:42:54.450 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 115 M 2026-02-24T15:42:54.450 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 974 M 2026-02-24T15:42:54.450 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:42:55.021 INFO:teuthology.orchestra.run.trial195.stdout:(1/42): ceph-20.3.0-5343.gd26e5c22.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-24T15:42:56.548 INFO:teuthology.orchestra.run.trial195.stdout:(2/42): ceph-mgr-20.3.0-5343.gd26e5c22.el9.x86_ 517 kB/s | 937 kB 00:01 2026-02-24T15:42:56.830 INFO:teuthology.orchestra.run.trial195.stdout:(3/42): ceph-osd-20.3.0-5343.gd26e5c22.el9.x86_ 579 kB/s | 163 kB 00:00 2026-02-24T15:42:59.618 INFO:teuthology.orchestra.run.trial195.stdout:(4/42): ceph-mds-20.3.0-5343.gd26e5c22.el9.x86_ 424 kB/s | 2.0 MB 00:04 2026-02-24T15:43:00.660 INFO:teuthology.orchestra.run.trial195.stdout:(5/42): ceph-mon-20.3.0-5343.gd26e5c22.el9.x86_ 884 kB/s | 4.9 MB 00:05 2026-02-24T15:43:01.010 INFO:teuthology.orchestra.run.trial195.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5343.gd26e 850 kB/s | 297 kB 00:00 2026-02-24T15:43:01.296 INFO:teuthology.orchestra.run.trial195.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 141 kB/s | 40 kB 00:00 2026-02-24T15:43:01.487 INFO:teuthology.orchestra.run.trial195.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 378 kB/s | 72 kB 00:00 2026-02-24T15:43:01.715 INFO:teuthology.orchestra.run.trial195.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2026-02-24T15:43:01.994 INFO:teuthology.orchestra.run.trial195.stdout:(10/42): python3-cryptography-36.0.1-5.el9.x86_ 4.5 MB/s | 1.2 MB 00:00 2026-02-24T15:43:02.075 INFO:teuthology.orchestra.run.trial195.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2026-02-24T15:43:02.157 INFO:teuthology.orchestra.run.trial195.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 135 kB 00:00 2026-02-24T15:43:02.240 INFO:teuthology.orchestra.run.trial195.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 1.8 MB/s | 150 kB 00:00 2026-02-24T15:43:02.322 INFO:teuthology.orchestra.run.trial195.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 1.5 MB/s | 126 kB 00:00 2026-02-24T15:43:02.408 INFO:teuthology.orchestra.run.trial195.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 2.5 MB/s | 218 kB 00:00 2026-02-24T15:43:02.492 INFO:teuthology.orchestra.run.trial195.stdout:(16/42): unzip-6.0-59.el9.x86_64.rpm 2.1 MB/s | 182 kB 00:00 2026-02-24T15:43:02.578 INFO:teuthology.orchestra.run.trial195.stdout:(17/42): zip-3.0-35.el9.x86_64.rpm 3.0 MB/s | 266 kB 00:00 2026-02-24T15:43:02.641 INFO:teuthology.orchestra.run.trial195.stdout:(18/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 3.9 MB/s | 246 kB 00:00 2026-02-24T15:43:02.658 INFO:teuthology.orchestra.run.trial195.stdout:(19/42): lua-5.4.4-4.el9.x86_64.rpm 11 MB/s | 188 kB 00:00 2026-02-24T15:43:02.674 INFO:teuthology.orchestra.run.trial195.stdout:(20/42): python3-libstoragemgmt-1.10.1-1.el9.x8 11 MB/s | 177 kB 00:00 2026-02-24T15:43:02.686 INFO:teuthology.orchestra.run.trial195.stdout:(21/42): python3-packaging-20.9-5.el9.noarch.rp 6.3 MB/s | 77 kB 00:00 2026-02-24T15:43:02.699 INFO:teuthology.orchestra.run.trial195.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 3.3 MB/s | 42 kB 00:00 2026-02-24T15:43:02.748 INFO:teuthology.orchestra.run.trial195.stdout:(23/42): lua-devel-5.4.4-4.el9.x86_64.rpm 456 kB/s | 22 kB 00:00 2026-02-24T15:43:03.156 INFO:teuthology.orchestra.run.trial195.stdout:(24/42): luarocks-3.9.2-5.el9.noarch.rpm 371 kB/s | 151 kB 00:00 2026-02-24T15:43:03.227 INFO:teuthology.orchestra.run.trial195.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 417 kB/s | 29 kB 00:00 2026-02-24T15:43:03.306 INFO:teuthology.orchestra.run.trial195.stdout:(26/42): python3-backports-tarfile-1.2.0-1.el9. 760 kB/s | 60 kB 00:00 2026-02-24T15:43:03.379 INFO:teuthology.orchestra.run.trial195.stdout:(27/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 597 kB/s | 43 kB 00:00 2026-02-24T15:43:03.488 INFO:teuthology.orchestra.run.trial195.stdout:(28/42): python3-cheroot-10.0.1-4.el9.noarch.rp 1.5 MB/s | 173 kB 00:00 2026-02-24T15:43:03.627 INFO:teuthology.orchestra.run.trial195.stdout:(29/42): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2026-02-24T15:43:03.688 INFO:teuthology.orchestra.run.trial195.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 175 kB/s | 11 kB 00:00 2026-02-24T15:43:03.750 INFO:teuthology.orchestra.run.trial195.stdout:(31/42): python3-jaraco-classes-3.2.1-5.el9.noa 287 kB/s | 18 kB 00:00 2026-02-24T15:43:03.812 INFO:teuthology.orchestra.run.trial195.stdout:(32/42): python3-jaraco-collections-3.0.0-8.el9 376 kB/s | 23 kB 00:00 2026-02-24T15:43:03.873 INFO:teuthology.orchestra.run.trial195.stdout:(33/42): python3-jaraco-context-6.0.1-3.el9.noa 319 kB/s | 20 kB 00:00 2026-02-24T15:43:03.935 INFO:teuthology.orchestra.run.trial195.stdout:(34/42): python3-jaraco-functools-3.5.0-2.el9.n 315 kB/s | 19 kB 00:00 2026-02-24T15:43:03.997 INFO:teuthology.orchestra.run.trial195.stdout:(35/42): python3-jaraco-text-4.0.0-2.el9.noarch 422 kB/s | 26 kB 00:00 2026-02-24T15:43:04.065 INFO:teuthology.orchestra.run.trial195.stdout:(36/42): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2026-02-24T15:43:04.127 INFO:teuthology.orchestra.run.trial195.stdout:(37/42): python3-portend-3.1.0-2.el9.noarch.rpm 266 kB/s | 16 kB 00:00 2026-02-24T15:43:04.194 INFO:teuthology.orchestra.run.trial195.stdout:(38/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2026-02-24T15:43:04.257 INFO:teuthology.orchestra.run.trial195.stdout:(39/42): python3-tempora-5.0.0-2.el9.noarch.rpm 571 kB/s | 36 kB 00:00 2026-02-24T15:43:04.319 INFO:teuthology.orchestra.run.trial195.stdout:(40/42): python3-zc-lockfile-2.0-10.el9.noarch. 325 kB/s | 20 kB 00:00 2026-02-24T15:43:17.199 INFO:teuthology.orchestra.run.trial195.stdout:(41/42): ceph-osd-classic-20.3.0-5343.gd26e5c22 742 kB/s | 15 MB 00:20 2026-02-24T15:45:32.452 INFO:teuthology.orchestra.run.trial195.stdout:(42/42): ceph-osd-crimson-20.3.0-5343.gd26e5c22 588 kB/s | 88 MB 02:32 2026-02-24T15:45:32.454 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:32.454 INFO:teuthology.orchestra.run.trial195.stdout:Total 748 kB/s | 115 MB 02:38 2026-02-24T15:45:32.575 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:45:32.594 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:45:32.594 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:45:32.994 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:45:32.994 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:45:33.424 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:45:33.425 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-24T15:45:33.431 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-24T15:45:33.436 INFO:teuthology.orchestra.run.trial195.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-24T15:45:33.438 INFO:teuthology.orchestra.run.trial195.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-24T15:45:33.444 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-24T15:45:33.467 INFO:teuthology.orchestra.run.trial195.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-24T15:45:33.471 INFO:teuthology.orchestra.run.trial195.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-24T15:45:33.475 INFO:teuthology.orchestra.run.trial195.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-24T15:45:33.480 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-24T15:45:33.486 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-24T15:45:33.490 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-24T15:45:33.496 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-24T15:45:33.501 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-24T15:45:33.505 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-24T15:45:33.510 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-24T15:45:33.514 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-24T15:45:33.520 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-24T15:45:33.524 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-24T15:45:33.547 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-24T15:45:33.559 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-24T15:45:33.564 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-24T15:45:33.574 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-24T15:45:33.581 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T15:45:33.594 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-24T15:45:33.610 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-24T15:45:33.685 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-24T15:45:33.693 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-24T15:45:33.716 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-24T15:45:33.745 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-24T15:45:33.751 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-24T15:45:33.802 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-24T15:45:33.843 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 32/42 2026-02-24T15:45:33.844 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T15:45:33.867 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T15:45:33.868 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:45:33.868 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T15:45:33.868 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T15:45:33.868 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T15:45:33.868 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:33.871 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-24T15:45:33.888 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:45:33.888 INFO:teuthology.orchestra.run.trial195.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T15:45:33.889 INFO:teuthology.orchestra.run.trial195.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T15:45:33.889 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:33.895 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:45:33.918 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:45:33.918 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T15:45:33.918 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:34.515 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-24T15:45:39.937 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 37/42 2026-02-24T15:45:39.947 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 38/42 2026-02-24T15:45:39.948 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T15:45:39.971 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T15:45:39.971 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:45:39.971 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T15:45:39.972 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T15:45:39.972 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T15:45:39.972 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:40.176 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T15:45:40.197 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T15:45:40.197 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:45:40.197 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T15:45:40.197 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T15:45:40.197 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T15:45:40.197 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:40.279 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T15:45:40.299 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T15:45:40.299 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:45:40.299 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T15:45:40.299 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T15:45:40.299 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T15:45:40.300 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:40.300 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T15:45:40.306 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T15:45:40.329 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T15:45:41.228 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T15:45:41.228 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/42 2026-02-24T15:45:41.229 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/42 2026-02-24T15:45:41.229 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/42 2026-02-24T15:45:41.229 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/42 2026-02-24T15:45:41.229 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/42 2026-02-24T15:45:41.229 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/42 2026-02-24T15:45:41.229 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 7/42 2026-02-24T15:45:41.229 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 8/42 2026-02-24T15:45:41.230 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-24T15:45:41.230 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-24T15:45:41.230 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-24T15:45:41.230 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-24T15:45:41.230 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-24T15:45:41.230 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-24T15:45:41.230 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-24T15:45:41.231 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-24T15:45:41.232 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-24T15:45:41.233 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-24T15:45:41.234 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-24T15:45:41.287 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:41.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T15:45:41.289 INFO:teuthology.orchestra.run.trial195.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T15:45:41.290 INFO:teuthology.orchestra.run.trial195.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: zip-3.0-35.el9.x86_64 2026-02-24T15:45:41.291 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:41.292 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:45:41.333 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-base 2026-02-24T15:45:41.647 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:06:03 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:45:41.713 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:41.730 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:45:41.731 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:45:41.732 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:45:41.753 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install cephadm 2026-02-24T15:45:42.066 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:06:04 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:45:42.150 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:45:42.150 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout: cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 1.0 M 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 1.0 M 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 1.0 M 2026-02-24T15:45:42.151 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:45:44.165 INFO:teuthology.orchestra.run.trial195.stdout:cephadm-20.3.0-5343.gd26e5c22.el9.noarch.rpm 517 kB/s | 1.0 MB 00:02 2026-02-24T15:45:44.165 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:44.165 INFO:teuthology.orchestra.run.trial195.stdout:Total 516 kB/s | 1.0 MB 00:02 2026-02-24T15:45:44.165 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:45:44.168 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:45:44.168 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:45:44.172 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:45:44.172 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:45:44.191 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:45:44.252 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:45:44.266 INFO:teuthology.orchestra.run.trial195.stdout: Installing : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:45:44.333 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:45:44.393 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:45:44.393 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:44.393 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:45:44.394 INFO:teuthology.orchestra.run.trial195.stdout: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:45:44.394 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:44.394 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:45:44.420 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T15:45:44.736 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:06:06 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:45:44.819 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repo Size 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 149 k 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 149 k 2026-02-24T15:45:44.820 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 455 k 2026-02-24T15:45:44.821 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:45:45.727 INFO:teuthology.orchestra.run.trial195.stdout:ceph-immutable-object-cache-20.3.0-5343.gd26e5c 165 kB/s | 149 kB 00:00 2026-02-24T15:45:45.727 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:45:45.727 INFO:teuthology.orchestra.run.trial195.stdout:Total 165 kB/s | 149 kB 00:00 2026-02-24T15:45:45.727 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:45:45.732 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:45:45.732 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:45:45.758 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:45:45.758 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:45:45.808 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:45:45.809 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:45:45.832 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:45:45.832 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:45:45.832 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T15:45:45.832 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:46.124 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:45:46.125 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:46.125 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:45:46.125 INFO:teuthology.orchestra.run.trial195.stdout: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:45:46.125 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:46.125 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:45:46.153 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr 2026-02-24T15:45:46.471 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:06:08 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:45:46.537 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:45:46.555 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:45:46.555 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:45:46.555 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:45:46.577 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T15:45:46.893 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:06:08 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:45:46.978 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 51 M 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 57 k 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 17 k 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T15:45:46.979 INFO:teuthology.orchestra.run.trial195.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout:Install 13 Packages 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 55 M 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 262 M 2026-02-24T15:45:46.980 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:45:47.755 INFO:teuthology.orchestra.run.trial195.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5343.gd26 49 kB/s | 17 kB 00:00 2026-02-24T15:45:47.835 INFO:teuthology.orchestra.run.trial195.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5343.gd2 132 kB/s | 57 kB 00:00 2026-02-24T15:45:48.244 INFO:teuthology.orchestra.run.trial195.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 97 kB/s | 48 kB 00:00 2026-02-24T15:45:48.548 INFO:teuthology.orchestra.run.trial195.stdout:(4/13): python3-protobuf-3.14.0-17.el9.noarch.r 375 kB/s | 267 kB 00:00 2026-02-24T15:45:48.627 INFO:teuthology.orchestra.run.trial195.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 6.8 MB/s | 551 kB 00:00 2026-02-24T15:45:48.636 INFO:teuthology.orchestra.run.trial195.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 2.0 MB/s | 19 kB 00:00 2026-02-24T15:45:48.669 INFO:teuthology.orchestra.run.trial195.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 63 MB/s | 2.0 MB 00:00 2026-02-24T15:45:48.678 INFO:teuthology.orchestra.run.trial195.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 15 MB/s | 144 kB 00:00 2026-02-24T15:45:48.688 INFO:teuthology.orchestra.run.trial195.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 3.2 MB/s | 31 kB 00:00 2026-02-24T15:45:48.697 INFO:teuthology.orchestra.run.trial195.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 19 MB/s | 188 kB 00:00 2026-02-24T15:45:48.707 INFO:teuthology.orchestra.run.trial195.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 2.3 MB/s | 22 kB 00:00 2026-02-24T15:45:48.877 INFO:teuthology.orchestra.run.trial195.stdout:(12/13): protobuf-compiler-3.14.0-17.el9.x86_64 1.3 MB/s | 862 kB 00:00 2026-02-24T15:47:37.509 INFO:teuthology.orchestra.run.trial195.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5343.gd26e5c 471 kB/s | 51 MB 01:50 2026-02-24T15:47:37.512 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:47:37.512 INFO:teuthology.orchestra.run.trial195.stdout:Total 508 kB/s | 55 MB 01:50 2026-02-24T15:47:37.537 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:47:37.547 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:47:37.547 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:47:37.764 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:47:37.764 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:47:37.993 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:47:37.999 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-24T15:47:38.010 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-24T15:47:38.012 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-24T15:47:38.041 INFO:teuthology.orchestra.run.trial195.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-24T15:47:38.086 INFO:teuthology.orchestra.run.trial195.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-24T15:47:38.094 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-24T15:47:38.122 INFO:teuthology.orchestra.run.trial195.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-24T15:47:38.132 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-24T15:47:38.139 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-24T15:47:38.142 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-24T15:47:38.151 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 11/13 2026-02-24T15:47:41.062 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 12/13 2026-02-24T15:47:41.068 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T15:47:41.355 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T15:47:41.355 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 1/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 2/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 3/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-24T15:47:41.356 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-24T15:47:41.415 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T15:47:41.416 INFO:teuthology.orchestra.run.trial195.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T15:47:41.417 INFO:teuthology.orchestra.run.trial195.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T15:47:41.417 INFO:teuthology.orchestra.run.trial195.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T15:47:41.417 INFO:teuthology.orchestra.run.trial195.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T15:47:41.417 INFO:teuthology.orchestra.run.trial195.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T15:47:41.417 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:41.417 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:47:41.450 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T15:47:41.767 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:03 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:47:41.852 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout:======================================================================================= 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout:======================================================================================= 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 7.4 M 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T15:47:41.854 INFO:teuthology.orchestra.run.trial195.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:47:41.855 INFO:teuthology.orchestra.run.trial195.stdout:======================================================================================= 2026-02-24T15:47:41.856 INFO:teuthology.orchestra.run.trial195.stdout:Install 12 Packages 2026-02-24T15:47:41.856 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:41.856 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 43 M 2026-02-24T15:47:41.856 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 234 M 2026-02-24T15:47:41.856 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:47:42.265 INFO:teuthology.orchestra.run.trial195.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 4.1 MB/s | 184 kB 00:00 2026-02-24T15:47:42.283 INFO:teuthology.orchestra.run.trial195.stdout:(2/12): libgfortran-11.5.0-14.el9.x86_64.rpm 13 MB/s | 794 kB 00:00 2026-02-24T15:47:42.626 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] flexiblas-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:47:42.640 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] flexiblas-netlib-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:47:43.035 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] flexiblas-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:47:43.047 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] flexiblas-netlib-3.0.4-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:47:43.593 INFO:teuthology.orchestra.run.trial195.stdout:(3/12): flexiblas-3.0.4-9.el9.x86_64.rpm 22 kB/s | 30 kB 00:01 2026-02-24T15:47:44.013 INFO:teuthology.orchestra.run.trial195.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 35 kB/s | 15 kB 00:00 2026-02-24T15:47:44.095 INFO:teuthology.orchestra.run.trial195.stdout:(5/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 1.6 MB/s | 3.0 MB 00:01 2026-02-24T15:47:44.191 INFO:teuthology.orchestra.run.trial195.stdout:(6/12): openblas-0.3.29-1.el9.x86_64.rpm 237 kB/s | 42 kB 00:00 2026-02-24T15:47:44.526 INFO:teuthology.orchestra.run.trial195.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 728 kB/s | 244 kB 00:00 2026-02-24T15:47:46.388 INFO:teuthology.orchestra.run.trial195.stdout:(8/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 2.3 MB/s | 5.3 MB 00:02 2026-02-24T15:47:46.489 INFO:teuthology.orchestra.run.trial195.stdout:(9/12): python3-numpy-f2py-1.23.5-2.el9.x86_64. 4.3 MB/s | 442 kB 00:00 2026-02-24T15:47:47.801 INFO:teuthology.orchestra.run.trial195.stdout:(10/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 15 MB/s | 19 MB 00:01 2026-02-24T15:47:51.623 INFO:teuthology.orchestra.run.trial195.stdout:(11/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 885 kB/s | 6.1 MB 00:07 2026-02-24T15:47:54.725 INFO:teuthology.orchestra.run.trial195.stdout:(12/12): ceph-mgr-diskprediction-local-20.3.0-5 605 kB/s | 7.4 MB 00:12 2026-02-24T15:47:54.728 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:47:54.729 INFO:teuthology.orchestra.run.trial195.stdout:Total 3.3 MB/s | 43 MB 00:12 2026-02-24T15:47:54.887 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:47:54.900 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:47:54.900 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:47:55.102 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:47:55.103 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:47:55.321 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:47:55.324 INFO:teuthology.orchestra.run.trial195.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-24T15:47:55.330 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-24T15:47:55.364 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-24T15:47:55.368 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-24T15:47:55.428 INFO:teuthology.orchestra.run.trial195.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-24T15:47:55.430 INFO:teuthology.orchestra.run.trial195.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-24T15:47:55.451 INFO:teuthology.orchestra.run.trial195.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-24T15:47:55.774 INFO:teuthology.orchestra.run.trial195.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-24T15:47:55.847 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-24T15:47:56.515 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T15:47:56.974 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-24T15:47:56.975 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 1/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-24T15:47:57.267 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-24T15:47:57.268 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T15:47:57.268 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-24T15:47:57.330 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-24T15:47:57.330 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:57.330 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:47:57.330 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:47:57.330 INFO:teuthology.orchestra.run.trial195.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:57.331 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:47:57.370 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-rook 2026-02-24T15:47:57.693 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:19 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:47:57.776 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:47:57.777 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:47:57.777 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:47:57.777 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:47:57.777 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:47:57.777 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-rook noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 50 k 2026-02-24T15:47:57.777 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:57.778 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:47:57.779 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:47:57.779 INFO:teuthology.orchestra.run.trial195.stdout:Install 10 Packages 2026-02-24T15:47:57.779 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:57.780 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 2.0 M 2026-02-24T15:47:57.780 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 25 M 2026-02-24T15:47:57.780 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:47:58.449 INFO:teuthology.orchestra.run.trial195.stdout:(1/10): ceph-mgr-rook-20.3.0-5343.gd26e5c22.el9 101 kB/s | 50 kB 00:00 2026-02-24T15:47:58.678 INFO:teuthology.orchestra.run.trial195.stdout:(2/10): python3-pyasn1-0.4.8-7.el9.noarch.rpm 217 kB/s | 157 kB 00:00 2026-02-24T15:47:58.708 INFO:teuthology.orchestra.run.trial195.stdout:(3/10): python3-cachetools-4.2.4-1.el9.noarch.r 1.1 MB/s | 32 kB 00:00 2026-02-24T15:47:58.715 INFO:teuthology.orchestra.run.trial195.stdout:(4/10): python3-certifi-2023.05.07-4.el9.noarch 1.8 MB/s | 14 kB 00:00 2026-02-24T15:47:58.737 INFO:teuthology.orchestra.run.trial195.stdout:(5/10): python3-pyasn1-modules-0.4.8-7.el9.noar 353 kB/s | 277 kB 00:00 2026-02-24T15:47:58.738 INFO:teuthology.orchestra.run.trial195.stdout:(6/10): python3-google-auth-2.45.0-1.el9.noarch 11 MB/s | 254 kB 00:00 2026-02-24T15:47:58.747 INFO:teuthology.orchestra.run.trial195.stdout:(7/10): python3-rsa-4.9-2.el9.noarch.rpm 6.9 MB/s | 59 kB 00:00 2026-02-24T15:47:58.756 INFO:teuthology.orchestra.run.trial195.stdout:(8/10): python3-websocket-client-1.2.3-2.el9.no 9.8 MB/s | 90 kB 00:00 2026-02-24T15:47:58.790 INFO:teuthology.orchestra.run.trial195.stdout:(9/10): python3-kubernetes-26.1.0-3.el9.noarch. 19 MB/s | 1.0 MB 00:00 2026-02-24T15:47:59.045 INFO:teuthology.orchestra.run.trial195.stdout:(10/10): python3-requests-oauthlib-1.3.0-12.el9 90 kB/s | 54 kB 00:00 2026-02-24T15:47:59.046 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:47:59.046 INFO:teuthology.orchestra.run.trial195.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2026-02-24T15:47:59.073 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:47:59.081 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:47:59.081 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:47:59.120 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:47:59.121 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:47:59.196 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:47:59.247 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 1/10 2026-02-24T15:47:59.260 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 2/10 2026-02-24T15:47:59.275 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-24T15:47:59.279 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-24T15:47:59.287 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-24T15:47:59.327 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-24T15:47:59.339 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T15:47:59.658 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-24T15:47:59.673 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-24T15:47:59.674 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 1/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 2/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 3/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-24T15:47:59.792 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:47:59.855 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:47:59.886 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T15:48:00.201 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:22 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:00.286 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:00.287 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 184 k 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T15:48:00.288 INFO:teuthology.orchestra.run.trial195.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout:Install 7 Packages 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 7.1 M 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 34 M 2026-02-24T15:48:00.289 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:01.225 INFO:teuthology.orchestra.run.trial195.stdout:(1/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 328 kB/s | 249 kB 00:00 2026-02-24T15:48:01.325 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:48:01.446 INFO:teuthology.orchestra.run.trial195.stdout:(2/7): ceph-mgr-cephadm-20.3.0-5343.gd26e5c22.e 188 kB/s | 184 kB 00:00 2026-02-24T15:48:01.712 INFO:teuthology.orchestra.run.trial195.stdout:(3/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2026-02-24T15:48:01.728 INFO:teuthology.orchestra.run.trial195.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:48:01.744 INFO:teuthology.orchestra.run.trial195.stdout:(4/7): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2026-02-24T15:48:01.780 INFO:teuthology.orchestra.run.trial195.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 2.4 MB/s | 86 kB 00:00 2026-02-24T15:48:02.289 INFO:teuthology.orchestra.run.trial195.stdout:(6/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 33 kB/s | 35 kB 00:01 2026-02-24T15:48:04.644 INFO:teuthology.orchestra.run.trial195.stdout:(7/7): python3-babel-2.9.1-2.el9.noarch.rpm 1.4 MB/s | 6.0 MB 00:04 2026-02-24T15:48:04.646 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:04.646 INFO:teuthology.orchestra.run.trial195.stdout:Total 1.6 MB/s | 7.1 MB 00:04 2026-02-24T15:48:04.683 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:04.690 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:04.690 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:04.763 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:04.764 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:04.855 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:04.904 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-24T15:48:04.914 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-24T15:48:04.920 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-24T15:48:05.142 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T15:48:05.165 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-24T15:48:05.190 INFO:teuthology.orchestra.run.trial195.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-24T15:48:05.191 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T15:48:05.306 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T15:48:05.307 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/7 2026-02-24T15:48:05.307 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-24T15:48:05.307 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-24T15:48:05.307 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T15:48:05.307 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-24T15:48:05.307 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T15:48:05.369 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:05.370 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:05.401 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-osd 2026-02-24T15:48:05.716 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:27 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:05.782 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:05.801 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:05.801 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:05.801 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:05.823 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-osd-classic 2026-02-24T15:48:06.138 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:28 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:06.204 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:06.223 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:06.224 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:06.224 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:06.245 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-fuse 2026-02-24T15:48:06.557 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:28 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:06.644 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:06.644 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:06.644 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repo Size 2026-02-24T15:48:06.644 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:06.644 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:06.644 INFO:teuthology.orchestra.run.trial195.stdout: ceph-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 914 k 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 914 k 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 2.9 M 2026-02-24T15:48:06.645 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:08.218 INFO:teuthology.orchestra.run.trial195.stdout:ceph-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 581 kB/s | 914 kB 00:01 2026-02-24T15:48:08.218 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:08.218 INFO:teuthology.orchestra.run.trial195.stdout:Total 580 kB/s | 914 kB 00:01 2026-02-24T15:48:08.218 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:08.223 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:08.223 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:08.256 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:08.256 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:08.346 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:08.364 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:08.583 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:08.645 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:08.645 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:08.645 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:08.645 INFO:teuthology.orchestra.run.trial195.stdout: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:48:08.645 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:08.645 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:08.672 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-volume 2026-02-24T15:48:08.992 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:30 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:09.076 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout: ceph-volume noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 301 k 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:Install 2 Packages 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 652 k 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 2.2 M 2026-02-24T15:48:09.077 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:09.317 INFO:teuthology.orchestra.run.trial195.stdout:(1/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.3 MB/s | 351 kB 00:00 2026-02-24T15:48:09.803 INFO:teuthology.orchestra.run.trial195.stdout:(2/2): ceph-volume-20.3.0-5343.gd26e5c22.el9.no 474 kB/s | 301 kB 00:00 2026-02-24T15:48:09.803 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:09.804 INFO:teuthology.orchestra.run.trial195.stdout:Total 899 kB/s | 652 kB 00:00 2026-02-24T15:48:09.813 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:09.819 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:09.819 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:09.862 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:09.863 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:09.944 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:10.008 INFO:teuthology.orchestra.run.trial195.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-24T15:48:10.010 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T15:48:10.025 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T15:48:10.025 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:48:10.025 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T15:48:10.025 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:10.254 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 1/2 2026-02-24T15:48:10.316 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-24T15:48:10.317 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:10.317 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:10.317 INFO:teuthology.orchestra.run.trial195.stdout: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T15:48:10.317 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:10.317 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:10.345 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install librados-devel 2026-02-24T15:48:10.665 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:32 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:10.751 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:10.751 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:10.751 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repo Size 2026-02-24T15:48:10.751 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:10.751 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:10.751 INFO:teuthology.orchestra.run.trial195.stdout: librados-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 124 k 2026-02-24T15:48:10.752 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:10.752 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:10.752 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:10.752 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:48:10.758 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:10.758 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 124 k 2026-02-24T15:48:10.758 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 434 k 2026-02-24T15:48:10.758 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:11.797 INFO:teuthology.orchestra.run.trial195.stdout:librados-devel-20.3.0-5343.gd26e5c22.el9.x86_64 118 kB/s | 124 kB 00:01 2026-02-24T15:48:11.798 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:11.798 INFO:teuthology.orchestra.run.trial195.stdout:Total 118 kB/s | 124 kB 00:01 2026-02-24T15:48:11.798 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:11.801 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:11.801 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:11.831 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:11.832 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:11.887 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:11.903 INFO:teuthology.orchestra.run.trial195.stdout: Installing : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:11.990 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:12.052 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:12.053 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:12.053 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:12.053 INFO:teuthology.orchestra.run.trial195.stdout: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:48:12.053 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:12.053 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:12.082 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install libcephfs2 2026-02-24T15:48:12.394 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:34 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:12.460 INFO:teuthology.orchestra.run.trial195.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:12.479 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:12.479 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:12.479 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:12.501 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install libcephfs-devel 2026-02-24T15:48:12.817 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:34 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:12.903 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repo Size 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout: libcephfs-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 37 k 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 26 k 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:Install 2 Packages 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 63 k 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 227 k 2026-02-24T15:48:12.904 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:13.334 INFO:teuthology.orchestra.run.trial195.stdout:(1/2): libcephfs-devel-20.3.0-5343.gd26e5c22.el 87 kB/s | 37 kB 00:00 2026-02-24T15:48:13.335 INFO:teuthology.orchestra.run.trial195.stdout:(2/2): libcephfs-proxy2-20.3.0-5343.gd26e5c22.e 59 kB/s | 26 kB 00:00 2026-02-24T15:48:13.336 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:13.336 INFO:teuthology.orchestra.run.trial195.stdout:Total 145 kB/s | 63 kB 00:00 2026-02-24T15:48:13.336 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:13.339 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:13.340 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:13.371 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:13.371 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:13.420 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:13.420 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:48:13.435 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:48:13.452 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:48:13.513 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:48:13.514 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:48:13.576 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:48:13.577 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:13.577 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:13.577 INFO:teuthology.orchestra.run.trial195.stdout: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:48:13.577 INFO:teuthology.orchestra.run.trial195.stdout: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:48:13.577 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:13.577 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:13.603 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install librados2 2026-02-24T15:48:13.923 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:35 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:13.988 INFO:teuthology.orchestra.run.trial195.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:14.007 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:14.008 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:14.008 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:14.031 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install librbd1 2026-02-24T15:48:14.351 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:36 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:14.418 INFO:teuthology.orchestra.run.trial195.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:14.436 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:14.437 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:14.437 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:14.458 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-rados 2026-02-24T15:48:14.775 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:36 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:14.840 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:14.859 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:14.860 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:14.861 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:14.881 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-rgw 2026-02-24T15:48:15.197 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:37 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:15.263 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:15.282 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:15.282 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:15.282 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:15.303 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-cephfs 2026-02-24T15:48:15.618 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:37 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:15.684 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:15.702 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:15.703 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:15.703 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:15.724 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-rbd 2026-02-24T15:48:16.038 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:38 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:16.104 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:16.122 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:16.123 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:16.123 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:16.145 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install rbd-fuse 2026-02-24T15:48:16.465 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:38 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:16.549 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:16.549 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:16.549 INFO:teuthology.orchestra.run.trial195.stdout: Package Architecture Version Repository Size 2026-02-24T15:48:16.549 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:16.549 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:16.549 INFO:teuthology.orchestra.run.trial195.stdout: rbd-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 86 k 2026-02-24T15:48:16.549 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:16.550 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:16.550 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:16.550 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:48:16.550 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:16.550 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 86 k 2026-02-24T15:48:16.550 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 236 k 2026-02-24T15:48:16.550 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:17.119 INFO:teuthology.orchestra.run.trial195.stdout:rbd-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 152 kB/s | 86 kB 00:00 2026-02-24T15:48:17.120 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:17.120 INFO:teuthology.orchestra.run.trial195.stdout:Total 152 kB/s | 86 kB 00:00 2026-02-24T15:48:17.120 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:17.124 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:17.124 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:17.154 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:17.154 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:17.205 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:17.222 INFO:teuthology.orchestra.run.trial195.stdout: Installing : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:17.308 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:17.370 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:17.370 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:17.370 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:17.370 INFO:teuthology.orchestra.run.trial195.stdout: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:48:17.370 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:17.371 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:17.399 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install rbd-mirror 2026-02-24T15:48:17.716 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:39 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:17.801 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repo Size 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout: rbd-mirror x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.6 M 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 2.6 M 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 11 M 2026-02-24T15:48:17.802 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:22.827 INFO:teuthology.orchestra.run.trial195.stdout:rbd-mirror-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 528 kB/s | 2.6 MB 00:05 2026-02-24T15:48:22.827 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:22.827 INFO:teuthology.orchestra.run.trial195.stdout:Total 528 kB/s | 2.6 MB 00:05 2026-02-24T15:48:22.827 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:22.833 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:22.833 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:22.871 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:22.871 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:23.035 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:23.036 INFO:teuthology.orchestra.run.trial195.stdout: Installing : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:23.059 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:23.059 INFO:teuthology.orchestra.run.trial195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:48:23.059 INFO:teuthology.orchestra.run.trial195.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T15:48:23.059 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T15:48:23.059 INFO:teuthology.orchestra.run.trial195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T15:48:23.059 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:23.368 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:23.368 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:23.369 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:23.369 INFO:teuthology.orchestra.run.trial195.stdout: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:48:23.369 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:23.369 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:23.397 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install rbd-nbd 2026-02-24T15:48:23.716 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:45 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:23.802 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:23.802 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:23.802 INFO:teuthology.orchestra.run.trial195.stdout: Package Architecture Version Repository Size 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout: rbd-nbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 167 k 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 167 k 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 496 k 2026-02-24T15:48:23.803 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T15:48:24.374 INFO:teuthology.orchestra.run.trial195.stdout:rbd-nbd-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 291 kB/s | 167 kB 00:00 2026-02-24T15:48:24.374 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:48:24.374 INFO:teuthology.orchestra.run.trial195.stdout:Total 291 kB/s | 167 kB 00:00 2026-02-24T15:48:24.374 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T15:48:24.379 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T15:48:24.379 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T15:48:24.410 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T15:48:24.410 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T15:48:24.466 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T15:48:24.483 INFO:teuthology.orchestra.run.trial195.stdout: Installing : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:24.572 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:24.634 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:48:24.634 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:24.634 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T15:48:24.634 INFO:teuthology.orchestra.run.trial195.stdout: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:48:24.634 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:24.634 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:24.661 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-osd-crimson 2026-02-24T15:48:24.978 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:46 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:25.045 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:48:25.064 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:25.064 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T15:48:25.065 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T15:48:25.086 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-24T15:48:25.400 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:08:47 ago on Tue 24 Feb 2026 03:39:38 PM UTC. 2026-02-24T15:48:25.485 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repo Size 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 433 M 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T15:48:25.486 INFO:teuthology.orchestra.run.trial195.stdout:Install 1 Package 2026-02-24T15:48:25.487 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T15:48:25.487 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 433 M 2026-02-24T15:48:25.487 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 2.6 G 2026-02-24T15:48:25.487 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T16:08:17.275 INFO:teuthology.orchestra.run.trial195.stdout:ceph-osd-crimson-debuginfo-20.3.0-5343.gd26e5c2 372 kB/s | 433 MB 19:51 2026-02-24T16:08:17.277 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:08:17.277 INFO:teuthology.orchestra.run.trial195.stdout:Total 372 kB/s | 433 MB 19:51 2026-02-24T16:08:17.277 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T16:08:17.279 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T16:08:17.279 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T16:08:18.573 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T16:08:18.573 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T16:08:42.161 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T16:08:42.178 INFO:teuthology.orchestra.run.trial195.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:08:44.100 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:08:44.162 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:08:44.162 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:08:44.162 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T16:08:44.162 INFO:teuthology.orchestra.run.trial195.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:08:44.163 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:08:44.163 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:08:44.232 DEBUG:teuthology.parallel:result is None 2026-02-24T16:08:44.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:08:44.396 DEBUG:teuthology.orchestra.run.trial195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:08:44.417 INFO:teuthology.orchestra.run.trial195.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:08:44.417 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:08:44.417 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:08:44.419 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-24T16:08:44.419 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:08:44.419 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:08:44.484 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-24T16:08:44.484 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:08:44.484 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:08:44.548 DEBUG:teuthology.orchestra.run.trial195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:08:44.611 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-24T16:08:44.612 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:08:44.612 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:08:44.675 DEBUG:teuthology.orchestra.run.trial195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:08:44.739 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-24T16:08:44.740 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:08:44.740 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:08:44.803 DEBUG:teuthology.orchestra.run.trial195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:08:44.867 INFO:teuthology.run_tasks:Running task pexec... 2026-02-24T16:08:44.890 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-24T16:08:44.891 DEBUG:teuthology.orchestra.run.trial195:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-24T16:08:44.910 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:08:44.910 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-24T16:08:44.961 DEBUG:teuthology.parallel:result is None 2026-02-24T16:08:44.961 INFO:teuthology.run_tasks:Running task install... 2026-02-24T16:08:44.975 DEBUG:teuthology.task.install:project ceph 2026-02-24T16:08:44.975 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T16:08:44.975 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T16:08:44.975 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T16:08:44.984 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T16:08:44.984 INFO:teuthology.task.install:extra packages: [] 2026-02-24T16:08:44.985 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T16:08:44.985 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:08:45.139 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T16:08:45.139 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T16:08:45.535 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:08:45.535 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:08:45.536 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:08:45.561 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-24T16:08:45.562 DEBUG:teuthology.orchestra.run.trial195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:08:45.627 DEBUG:teuthology.orchestra.run.trial195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:08:45.691 DEBUG:teuthology.orchestra.run.trial195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:08:45.747 INFO:teuthology.orchestra.run.trial195.stdout:check_obsoletes = 1 2026-02-24T16:08:45.757 DEBUG:teuthology.orchestra.run.trial195:> sudo yum clean all 2026-02-24T16:08:45.919 INFO:teuthology.orchestra.run.trial195.stdout:56 files removed 2026-02-24T16:08:45.935 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-radosgw 2026-02-24T16:08:47.229 INFO:teuthology.orchestra.run.trial195.stdout:ceph packages for x86_64 80 kB/s | 92 kB 00:01 2026-02-24T16:08:47.900 INFO:teuthology.orchestra.run.trial195.stdout:ceph noarch packages 37 kB/s | 24 kB 00:00 2026-02-24T16:08:48.509 INFO:teuthology.orchestra.run.trial195.stdout:ceph source packages 3.5 kB/s | 2.1 kB 00:00 2026-02-24T16:08:49.365 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-24T16:08:52.463 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-24T16:08:55.394 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - CRB 16 MB/s | 8.0 MB 00:00 2026-02-24T16:08:57.174 INFO:teuthology.orchestra.run.trial195.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 20 kB 00:01 2026-02-24T16:08:58.429 INFO:teuthology.orchestra.run.trial195.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-24T16:09:01.700 INFO:teuthology.orchestra.run.trial195.stdout:lab-extras 3.1 MB/s | 50 kB 00:00 2026-02-24T16:09:02.879 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:02.898 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:02.898 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:02.899 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:02.922 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-test 2026-02-24T16:09:03.236 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:03.304 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:03.323 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:03.324 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:03.324 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:03.345 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph 2026-02-24T16:09:03.664 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:03.729 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:03.748 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:03.749 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:03.749 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:03.770 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-base 2026-02-24T16:09:04.086 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:04.152 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:04.171 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:04.172 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:04.172 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:04.193 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install cephadm 2026-02-24T16:09:04.510 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:04.576 INFO:teuthology.orchestra.run.trial195.stdout:Package cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:09:04.595 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:04.595 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:04.596 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:04.617 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:09:04.930 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:04.996 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:05.015 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:05.015 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:05.015 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:05.037 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr 2026-02-24T16:09:05.356 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:05.422 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:05.441 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:05.441 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:05.442 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:05.464 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:09:05.776 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:05.842 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:09:05.861 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:05.862 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:05.862 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:05.883 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:09:06.206 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:06.272 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:09:06.291 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:06.291 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:06.291 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:06.313 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:09:06.629 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:06.694 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:09:06.713 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:06.714 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:06.714 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:06.735 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:09:07.054 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:07.120 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:09:07.139 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:07.139 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:07.140 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:07.161 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-osd 2026-02-24T16:09:07.479 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:07.545 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:07.564 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:07.565 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:07.565 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:07.586 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-osd-classic 2026-02-24T16:09:07.904 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:07.971 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:07.990 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:07.990 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:07.990 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:08.013 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-fuse 2026-02-24T16:09:08.330 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:08.395 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:08.415 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:08.415 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:08.415 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:08.437 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install ceph-volume 2026-02-24T16:09:08.754 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:08.819 INFO:teuthology.orchestra.run.trial195.stdout:Package ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:09:08.838 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:08.839 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:08.839 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:08.860 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install librados-devel 2026-02-24T16:09:09.177 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:09.243 INFO:teuthology.orchestra.run.trial195.stdout:Package librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:09.262 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:09.263 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:09.263 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:09.284 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install libcephfs2 2026-02-24T16:09:09.604 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:09.670 INFO:teuthology.orchestra.run.trial195.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:09.690 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:09.691 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:09.691 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:09.712 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install libcephfs-devel 2026-02-24T16:09:10.030 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:10.096 INFO:teuthology.orchestra.run.trial195.stdout:Package libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:10.115 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:10.116 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:10.116 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:10.138 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install librados2 2026-02-24T16:09:10.460 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:10.526 INFO:teuthology.orchestra.run.trial195.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:10.545 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:10.546 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:10.546 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:10.568 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install librbd1 2026-02-24T16:09:10.885 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:10.951 INFO:teuthology.orchestra.run.trial195.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:10.969 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:10.970 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:10.970 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:10.992 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-rados 2026-02-24T16:09:11.312 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:11.377 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:11.396 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:11.397 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:11.397 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:11.420 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-rgw 2026-02-24T16:09:11.737 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:11.803 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:11.822 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:11.822 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:11.822 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:11.844 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-cephfs 2026-02-24T16:09:12.166 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:12.232 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:12.251 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:12.251 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:12.251 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:12.273 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install python3-rbd 2026-02-24T16:09:12.593 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:12.658 INFO:teuthology.orchestra.run.trial195.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:12.677 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:12.678 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:12.678 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:12.699 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install rbd-fuse 2026-02-24T16:09:13.018 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:13.084 INFO:teuthology.orchestra.run.trial195.stdout:Package rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:13.103 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:13.104 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:13.104 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:13.126 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install rbd-mirror 2026-02-24T16:09:13.446 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:13.512 INFO:teuthology.orchestra.run.trial195.stdout:Package rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:13.531 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:13.531 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:13.531 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:13.555 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install rbd-nbd 2026-02-24T16:09:13.874 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:09:13.941 INFO:teuthology.orchestra.run.trial195.stdout:Package rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:09:13.960 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:09:13.960 INFO:teuthology.orchestra.run.trial195.stdout:Nothing to do. 2026-02-24T16:09:13.960 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:09:13.986 DEBUG:teuthology.parallel:result is None 2026-02-24T16:09:13.986 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:09:14.156 DEBUG:teuthology.orchestra.run.trial195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:09:14.177 INFO:teuthology.orchestra.run.trial195.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:09:14.177 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:09:14.177 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:09:14.179 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-24T16:09:14.179 INFO:teuthology.run_tasks:Running task ceph... 2026-02-24T16:09:14.277 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-24T16:09:14.277 DEBUG:teuthology.orchestra.run.trial195:> sudo chmod 777 /var/log/ceph 2026-02-24T16:09:14.334 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-24T16:09:14.334 DEBUG:teuthology.orchestra.run.trial195:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-24T16:09:14.398 INFO:tasks.ceph:Creating extra log directories... 2026-02-24T16:09:14.399 DEBUG:teuthology.orchestra.run.trial195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-24T16:09:14.473 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-24T16:09:14.474 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-24T16:09:14.474 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68175', 'branch': 'wip-liucm-shard-redesign-crimson-only', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/fio_4M_rand_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '68175', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-liucm-shard-redesign-crimson-only'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'workunit': {'branch': 'wip-liucm-shard-redesign-crimson-only', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'owner': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'priority': 120, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9834, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-liucm-shard-redesign-crimson-only', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'targets': {'trial195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNs2UmTz9GKjdOR/PSlituT9KkcsRPvZIq7MOfrVjnXdYa45Vw1aHHseeIGSVGD+RaG7gEjWuf1pguoNch9sx48='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-24_15:28:57', 'tube': 'trial', 'user': 'liucm', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-24T16:09:14.475 DEBUG:teuthology.orchestra.run.trial195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-24T16:09:14.519 DEBUG:teuthology.orchestra.run.trial195:> sudo install -d -m0777 -- /var/run/ceph 2026-02-24T16:09:14.584 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:14.584 DEBUG:teuthology.orchestra.run.trial195:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:09:14.641 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:09:14.642 DEBUG:teuthology.orchestra.run.trial195:> stat /dev/vg_nvme/lv_1 2026-02-24T16:09:14.698 INFO:teuthology.orchestra.run.trial195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:09:14.698 INFO:teuthology.orchestra.run.trial195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:09:14.698 INFO:teuthology.orchestra.run.trial195.stdout:Device: 6h/6d Inode: 962 Links: 1 2026-02-24T16:09:14.698 INFO:teuthology.orchestra.run.trial195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:09:14.698 INFO:teuthology.orchestra.run.trial195.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:09:14.699 INFO:teuthology.orchestra.run.trial195.stdout:Access: 2026-02-24 15:48:23.204413937 +0000 2026-02-24T16:09:14.699 INFO:teuthology.orchestra.run.trial195.stdout:Modify: 2026-02-24 15:39:09.008485399 +0000 2026-02-24T16:09:14.699 INFO:teuthology.orchestra.run.trial195.stdout:Change: 2026-02-24 15:39:09.008485399 +0000 2026-02-24T16:09:14.699 INFO:teuthology.orchestra.run.trial195.stdout: Birth: 2026-02-24 15:39:09.008485399 +0000 2026-02-24T16:09:14.699 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:09:14.761 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records in 2026-02-24T16:09:14.761 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records out 2026-02-24T16:09:14.762 INFO:teuthology.orchestra.run.trial195.stderr:512 bytes copied, 0.00015104 s, 3.4 MB/s 2026-02-24T16:09:14.763 DEBUG:teuthology.orchestra.run.trial195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:09:14.820 DEBUG:teuthology.orchestra.run.trial195:> stat /dev/vg_nvme/lv_2 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout:Device: 6h/6d Inode: 986 Links: 1 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout:Access: 2026-02-24 15:48:23.204413937 +0000 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout:Modify: 2026-02-24 15:39:09.216478102 +0000 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout:Change: 2026-02-24 15:39:09.216478102 +0000 2026-02-24T16:09:14.875 INFO:teuthology.orchestra.run.trial195.stdout: Birth: 2026-02-24 15:39:09.216478102 +0000 2026-02-24T16:09:14.876 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:09:14.938 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records in 2026-02-24T16:09:14.939 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records out 2026-02-24T16:09:14.939 INFO:teuthology.orchestra.run.trial195.stderr:512 bytes copied, 0.0001401 s, 3.7 MB/s 2026-02-24T16:09:14.940 DEBUG:teuthology.orchestra.run.trial195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:09:14.997 DEBUG:teuthology.orchestra.run.trial195:> stat /dev/vg_nvme/lv_3 2026-02-24T16:09:15.054 INFO:teuthology.orchestra.run.trial195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:09:15.054 INFO:teuthology.orchestra.run.trial195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:09:15.054 INFO:teuthology.orchestra.run.trial195.stdout:Device: 6h/6d Inode: 1017 Links: 1 2026-02-24T16:09:15.054 INFO:teuthology.orchestra.run.trial195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:09:15.054 INFO:teuthology.orchestra.run.trial195.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:09:15.055 INFO:teuthology.orchestra.run.trial195.stdout:Access: 2026-02-24 15:48:23.204413937 +0000 2026-02-24T16:09:15.055 INFO:teuthology.orchestra.run.trial195.stdout:Modify: 2026-02-24 15:39:09.432470529 +0000 2026-02-24T16:09:15.055 INFO:teuthology.orchestra.run.trial195.stdout:Change: 2026-02-24 15:39:09.432470529 +0000 2026-02-24T16:09:15.055 INFO:teuthology.orchestra.run.trial195.stdout: Birth: 2026-02-24 15:39:09.432470529 +0000 2026-02-24T16:09:15.055 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:09:15.117 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records in 2026-02-24T16:09:15.117 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records out 2026-02-24T16:09:15.117 INFO:teuthology.orchestra.run.trial195.stderr:512 bytes copied, 0.00014542 s, 3.5 MB/s 2026-02-24T16:09:15.118 DEBUG:teuthology.orchestra.run.trial195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:09:15.175 DEBUG:teuthology.orchestra.run.trial195:> stat /dev/vg_nvme/lv_4 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout:Device: 6h/6d Inode: 1045 Links: 1 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout:Access: 2026-02-24 15:48:23.205413922 +0000 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout:Modify: 2026-02-24 15:39:09.645463061 +0000 2026-02-24T16:09:15.232 INFO:teuthology.orchestra.run.trial195.stdout:Change: 2026-02-24 15:39:09.645463061 +0000 2026-02-24T16:09:15.233 INFO:teuthology.orchestra.run.trial195.stdout: Birth: 2026-02-24 15:39:09.645463061 +0000 2026-02-24T16:09:15.233 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:09:15.294 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records in 2026-02-24T16:09:15.294 INFO:teuthology.orchestra.run.trial195.stderr:1+0 records out 2026-02-24T16:09:15.294 INFO:teuthology.orchestra.run.trial195.stderr:512 bytes copied, 0.000133407 s, 3.8 MB/s 2026-02-24T16:09:15.295 DEBUG:teuthology.orchestra.run.trial195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:09:15.352 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:09:15.352 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial195.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:09:15.352 INFO:tasks.ceph:Generating config... 2026-02-24T16:09:15.355 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-24T16:09:15.355 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-24T16:09:15.355 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-24T16:09:15.355 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-24T16:09:15.355 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-24T16:09:15.355 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-24T16:09:15.356 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-24T16:09:15.357 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-24T16:09:15.357 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-24T16:09:15.357 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-24T16:09:15.357 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-24T16:09:15.357 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-24T16:09:15.357 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-24T16:09:15.357 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-24T16:09:15.357 INFO:tasks.ceph:[global] auth client required = none 2026-02-24T16:09:15.357 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-24T16:09:15.357 INFO:tasks.ceph:[global] auth service required = none 2026-02-24T16:09:15.357 INFO:tasks.ceph:[global] auth supported = none 2026-02-24T16:09:15.357 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-24T16:09:15.357 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-24T16:09:15.358 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-24T16:09:15.359 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-24T16:09:15.360 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-24T16:09:15.361 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-24T16:09:15.362 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-24T16:09:15.362 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-24T16:09:15.362 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-24T16:09:15.363 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-24T16:09:15.363 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-24T16:09:15.363 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-24T16:09:15.363 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-24T16:09:15.363 INFO:tasks.ceph:Setting up mon.a... 2026-02-24T16:09:15.363 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-24T16:09:15.434 INFO:teuthology.orchestra.run.trial195.stdout:creating /etc/ceph/ceph.keyring 2026-02-24T16:09:15.437 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-24T16:09:15.521 DEBUG:teuthology.orchestra.run.trial195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:09:15.546 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.195')] 2026-02-24T16:09:15.547 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.195', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-24T16:09:15.549 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:15.549 DEBUG:teuthology.orchestra.run.trial195:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:09:15.603 DEBUG:teuthology.orchestra.run.trial195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.195 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:09:15.684 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:15.681+0000 7fc194e45f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:15.684 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:15.681+0000 7fc194e45f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:15.684 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:15.681+0000 7fc194e45f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:monmaptool: generated fsid 8790c0b4-9560-4c5c-836a-a19ed753b087 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:setting min_mon_release = tentacle 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:epoch 0 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:fsid 8790c0b4-9560-4c5c-836a-a19ed753b087 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:last_changed 2026-02-24T16:09:15.683190+0000 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:created 2026-02-24T16:09:15.683190+0000 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:min_mon_release 20 (tentacle) 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:election_strategy: 1 2026-02-24T16:09:15.686 INFO:teuthology.orchestra.run.trial195.stdout:0: [v2:10.20.193.195:3300/0,v1:10.20.193.195:6789/0] mon.a 2026-02-24T16:09:15.687 INFO:teuthology.orchestra.run.trial195.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-24T16:09:15.687 DEBUG:teuthology.orchestra.run.trial195:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:09:15.742 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8790c0b4-9560-4c5c-836a-a19ed753b087... 2026-02-24T16:09:15.744 DEBUG:teuthology.orchestra.run.trial195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-24T16:09:15.824 INFO:teuthology.orchestra.run.trial195.stdout:[global] 2026-02-24T16:09:15.824 INFO:teuthology.orchestra.run.trial195.stdout: chdir = "" 2026-02-24T16:09:15.824 INFO:teuthology.orchestra.run.trial195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-24T16:09:15.824 INFO:teuthology.orchestra.run.trial195.stdout: auth supported = none 2026-02-24T16:09:15.824 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: filestore xattr use omap = true 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: mon clock drift allowed = 1.000 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: osd crush chooseleaf type = 0 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: auth debug = true 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: ms die on old message = true 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: ms die on bug = true 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.825 INFO:teuthology.orchestra.run.trial195.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: mon pg warn max object skew = 0 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: # disable pg_autoscaler by default for new pools 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: osd pool default size = 2 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: mon osd allow primary affinity = true 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: mon osd allow pg remap = true 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: mon warn on legacy crush tunables = false 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: mon warn on crush straw calc version zero = false 2026-02-24T16:09:15.826 INFO:teuthology.orchestra.run.trial195.stdout: mon warn on no sortbitwise = false 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: mon warn on osd down out interval zero = false 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: mon warn on too few osds = false 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: mon_allow_pool_size_one = true 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.827 INFO:teuthology.orchestra.run.trial195.stdout: osd default data pool replay window = 5 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: mon allow pool delete = true 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: mon cluster log file level = debug 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: debug asserts on shutdown = true 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: mon health detail to clog = false 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: mon host = 10.20.193.195 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: auth client required = none 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: auth cluster required = none 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: auth service required = none 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: debug asok = 0/0 2026-02-24T16:09:15.828 INFO:teuthology.orchestra.run.trial195.stdout: debug auth = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug bdev = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug bluefs = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug bluestore = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug buffer = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug civetweb = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug client = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug compressor = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug context = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug crush = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug crypto = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug dpdk = 0/0 2026-02-24T16:09:15.829 INFO:teuthology.orchestra.run.trial195.stdout: debug eventtrace = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug filer = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug filestore = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug finisher = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug fuse = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug heartbeatmap = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug javaclient = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug journal = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug journaler = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug kstore = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug leveldb = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug lockdep = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug mds = 0/0 2026-02-24T16:09:15.830 INFO:teuthology.orchestra.run.trial195.stdout: debug mds balancer = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug mds locker = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug mds log = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug mds log expire = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug mds migrator = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug memdb = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug mgr = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug mgrc = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug mon = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug monc = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug ms = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug objclass = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug objectcacher = 0/0 2026-02-24T16:09:15.831 INFO:teuthology.orchestra.run.trial195.stdout: debug objecter = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug optracker = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug osd = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug paxos = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug perfcounter = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rados = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rbd = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rbd mirror = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rbd replay = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug refs = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rgw = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rgw lifecycle = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rgw notification = 0/0 2026-02-24T16:09:15.832 INFO:teuthology.orchestra.run.trial195.stdout: debug rgw sync = 0/0 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: debug rocksdb = 0/0 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: debug striper = 0/0 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: debug throttle = 0/0 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: debug timer = 0/0 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: debug tp = 0/0 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: ms client mode = crc 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: ms cluster mode = crc 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: ms mon client mode = crc 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: ms mon cluster mode = crc 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: ms mon service mode = crc 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: ms service mode = crc 2026-02-24T16:09:15.833 INFO:teuthology.orchestra.run.trial195.stdout: fsid = 8790c0b4-9560-4c5c-836a-a19ed753b087 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout:[osd] 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: osd journal size = 100 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: osd scrub load threshold = 5.0 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: osd scrub max interval = 600 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: osd mclock profile = high_recovery_ops 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: osd mclock skip benchmark = true 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: osd recover clone overlap = true 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: osd recovery max chunk = 1048576 2026-02-24T16:09:15.834 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd debug shutdown = true 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd debug op order = true 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd debug verify stray on activate = true 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd debug trim objects = true 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd open classes on start = true 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd debug pg log writeout = true 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd deep scrub update digest min age = 30 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: osd map max advance = 10 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.835 INFO:teuthology.orchestra.run.trial195.stdout: journal zero on create = true 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: filestore ondisk finisher threads = 3 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: filestore apply finisher threads = 3 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: bdev debug aio = true 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: osd debug misdirected ops = true 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: debug monc = 20 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: crimson cpu num = 4 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: crimson osd obc lru size = 10 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: debug filestore = 0/0 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: debug journal = 0/0 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: debug ms = 0/0 2026-02-24T16:09:15.836 INFO:teuthology.orchestra.run.trial195.stdout: debug osd = 0/0 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore backref = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore cache = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore cleaner = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore device = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore epm = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore fixedkv tree = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore journal = 20 2026-02-24T16:09:15.837 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore lba = 20 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore odata = 20 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore omap = 20 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore onode = 20 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore t = 20 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: debug seastore tm = 20 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: osd objectstore = seastore 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: osd shutdown pgref assert = True 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: seastore cachepin size pershard = 64M 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: seastore max concurrent transactions = 8 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout:[mgr] 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: debug ms = 1 2026-02-24T16:09:15.838 INFO:teuthology.orchestra.run.trial195.stdout: debug mgr = 20 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: debug mon = 20 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: debug auth = 20 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: mgr/telemetry/nag = false 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout:[mon] 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: debug ms = 0/0 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: debug mon = 0/0 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: debug paxos = 0/0 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: debug auth = 20 2026-02-24T16:09:15.839 INFO:teuthology.orchestra.run.trial195.stdout: mon data avail warn = 5 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: mon mgr mkfs grace = 240 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: mon osd reporter subtree level = osd 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: mon osd prime pg temp = true 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: auth mon ticket ttl = 660 # 11m 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: auth service ticket ttl = 240 # 4m 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: # don't complain about insecure global_id in the test suite 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-24T16:09:15.840 INFO:teuthology.orchestra.run.trial195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: # 1m isn't quite enough 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: mon_down_mkfs_grace = 2m 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: mon_warn_on_filestore_osds = false 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: osd pool default crimson = True 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout:[client] 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: rgw cache enabled = true 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: rgw enable ops log = true 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: rgw enable usage log = true 2026-02-24T16:09:15.841 INFO:teuthology.orchestra.run.trial195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-24T16:09:15.842 INFO:teuthology.orchestra.run.trial195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-24T16:09:15.842 INFO:teuthology.orchestra.run.trial195.stdout:[mon.a] 2026-02-24T16:09:15.843 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-24T16:09:15.843 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-24T16:09:15.934 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-24T16:09:15.934 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:15.934 DEBUG:teuthology.orchestra.run.trial195:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-24T16:09:15.950 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:15.951 DEBUG:teuthology.orchestra.run.trial195:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-24T16:09:16.006 INFO:tasks.ceph:Sending monmap to node ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:09:16.007 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:16.007 DEBUG:teuthology.orchestra.run.trial195:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-24T16:09:16.007 DEBUG:teuthology.orchestra.run.trial195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:09:16.080 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:16.080 DEBUG:teuthology.orchestra.run.trial195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:09:16.136 INFO:tasks.ceph:Setting up mon nodes... 2026-02-24T16:09:16.136 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-24T16:09:16.137 DEBUG:teuthology.orchestra.run.trial195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:09:16.227 INFO:teuthology.orchestra.run.trial195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:09:16.230 INFO:tasks.ceph:Setting up mds nodes... 2026-02-24T16:09:16.230 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-24T16:09:16.230 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-24T16:09:16.271 INFO:teuthology.orchestra.run.trial195.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-24T16:09:16.283 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-24T16:09:16.283 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial195.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:09:16.283 DEBUG:teuthology.orchestra.run.trial195:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-24T16:09:16.347 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:09:16.347 INFO:tasks.ceph:role: osd.0 2026-02-24T16:09:16.347 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:09:16.347 DEBUG:teuthology.orchestra.run.trial195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-24T16:09:16.411 INFO:teuthology.orchestra.run.trial195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:09:16.411 INFO:teuthology.orchestra.run.trial195.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:09:16.412 INFO:teuthology.orchestra.run.trial195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:09:16.986 INFO:teuthology.orchestra.run.trial195.stdout:Discarding blocks...Done. 2026-02-24T16:09:16.988 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial195.front.sepia.ceph.com -o noatime 2026-02-24T16:09:16.988 DEBUG:teuthology.orchestra.run.trial195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-24T16:09:17.058 DEBUG:teuthology.orchestra.run.trial195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-24T16:09:17.125 DEBUG:teuthology.orchestra.run.trial195:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-24T16:09:17.189 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:09:17.189 INFO:tasks.ceph:role: osd.1 2026-02-24T16:09:17.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:09:17.189 DEBUG:teuthology.orchestra.run.trial195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-24T16:09:17.255 INFO:teuthology.orchestra.run.trial195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:09:17.255 INFO:teuthology.orchestra.run.trial195.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:09:17.255 INFO:teuthology.orchestra.run.trial195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:09:17.255 INFO:teuthology.orchestra.run.trial195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:09:17.256 INFO:teuthology.orchestra.run.trial195.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:09:17.256 INFO:teuthology.orchestra.run.trial195.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:09:17.256 INFO:teuthology.orchestra.run.trial195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:09:17.256 INFO:teuthology.orchestra.run.trial195.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:09:17.256 INFO:teuthology.orchestra.run.trial195.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:09:17.256 INFO:teuthology.orchestra.run.trial195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:09:17.827 INFO:teuthology.orchestra.run.trial195.stdout:Discarding blocks...Done. 2026-02-24T16:09:17.829 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial195.front.sepia.ceph.com -o noatime 2026-02-24T16:09:17.829 DEBUG:teuthology.orchestra.run.trial195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-24T16:09:17.896 DEBUG:teuthology.orchestra.run.trial195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-24T16:09:17.963 DEBUG:teuthology.orchestra.run.trial195:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-24T16:09:18.028 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:09:18.028 INFO:tasks.ceph:role: osd.2 2026-02-24T16:09:18.028 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:09:18.029 DEBUG:teuthology.orchestra.run.trial195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-24T16:09:18.093 INFO:teuthology.orchestra.run.trial195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:09:18.094 INFO:teuthology.orchestra.run.trial195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:09:18.653 INFO:teuthology.orchestra.run.trial195.stdout:Discarding blocks...Done. 2026-02-24T16:09:18.655 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial195.front.sepia.ceph.com -o noatime 2026-02-24T16:09:18.656 DEBUG:teuthology.orchestra.run.trial195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-24T16:09:18.724 DEBUG:teuthology.orchestra.run.trial195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-24T16:09:18.792 DEBUG:teuthology.orchestra.run.trial195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:09:18.942 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:09:18.998 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:18,996 seastar - Reactor backend: linux-aio 2026-02-24T16:09:19.007 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:19,006 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:19.007 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:19,006 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:19.009 INFO:teuthology.orchestra.run.trial195.stderr:==39635==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:09:19.068 INFO:teuthology.orchestra.run.trial195.stderr:DEBUG 2026-02-24 16:09:19,067 [shard 0:main] seastar - reactor::drain 2026-02-24T16:09:21.164 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:21.164 INFO:teuthology.orchestra.run.trial195.stderr:Suppressions used: 2026-02-24T16:09:21.164 INFO:teuthology.orchestra.run.trial195.stderr: count bytes template 2026-02-24T16:09:21.165 INFO:teuthology.orchestra.run.trial195.stderr: 1 8 MallocExtension::Register 2026-02-24T16:09:21.165 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:21.165 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:09:21.183 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:09:21.183 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:09:21.231 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:09:21.231 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:09:21.238 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:21,237 seastar - Reactor backend: linux-aio 2026-02-24T16:09:21.252 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:21,250 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:21.252 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:21,251 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:21.257 INFO:teuthology.orchestra.run.trial195.stderr:==39633==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:09:21.257 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:21,255 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:09:21.415 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x157c7 0x10d1d95 0xf741d0 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.415 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10d1d95 0xf741d0 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.416 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9fe52cb 0x9fe6fa7 0xa0c11c2 0xa0c193b 0xa0c1e26 0xa0c21b9 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.448 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1fb10 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe4b44 0x9fe500e 0x9fe5281 0x9fe6280 0x9fe67ee 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:21.477 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 33 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119823 /lib64/libasan.so.8+0xfd239 /lib64/libasan.so.8+0xedcb5 0xe6ca27 0x9f2ee74 0x9f2efb8 0x9f2f09d 0x9f2f2f9 0x9f2f3ec 0x9f2f4c8 0x9f2fb84 0x9f2fceb 0x9f300ef 0x9f303ac 0x9f311f4 0xa0a9b3b 0xa0a9c35 0x7efee32 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.510 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xe6c83b 0xe6cd7e 0x77b1595 0x77b1696 0x763e27c 0x7efeae4 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.510 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191cbf /lib64/libasan.so.8+0xb9ea /lib64/libasan.so.8+0xbf98 /lib64/libasan.so.8+0xc08c /lib64/libasan.so.8+0xc360 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0xa0d88c9 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.510 INFO:teuthology.orchestra.run.trial195.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff90291052 0xffffffff90f50451 0xffffffff91000ba2 2026-02-24T16:09:21.535 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:21.595 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea71f 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.596 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.597 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ce97a1 0x80977d2 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.744 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xa0d667e 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.744 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d8 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe882e 0x9fe8adb 0xa0d8a91 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.744 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xedc3a 0xe6ca27 0x7bc7122 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.826 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x80977ef 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:21.862 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ceaa17 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.862 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2c95a6b 0x2cea88a 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:21.863 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea2cd 0x80977f4 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:24.026 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:24.026 INFO:teuthology.orchestra.run.trial195.stderr:Suppressions used: 2026-02-24T16:09:24.026 INFO:teuthology.orchestra.run.trial195.stderr: count bytes template 2026-02-24T16:09:24.026 INFO:teuthology.orchestra.run.trial195.stderr: 1 8 MallocExtension::Register 2026-02-24T16:09:24.026 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:24.027 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:09:24.065 DEBUG:teuthology.orchestra.run.trial195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-24T16:09:24.091 DEBUG:teuthology.orchestra.run.trial195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:09:24.244 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:09:24.296 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:24,295 seastar - Reactor backend: linux-aio 2026-02-24T16:09:24.307 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:24,305 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:24.307 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:24,305 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:24.309 INFO:teuthology.orchestra.run.trial195.stderr:==39702==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:09:24.368 INFO:teuthology.orchestra.run.trial195.stderr:DEBUG 2026-02-24 16:09:24,367 [shard 0:main] seastar - reactor::drain 2026-02-24T16:09:26.480 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:26.480 INFO:teuthology.orchestra.run.trial195.stderr:Suppressions used: 2026-02-24T16:09:26.481 INFO:teuthology.orchestra.run.trial195.stderr: count bytes template 2026-02-24T16:09:26.481 INFO:teuthology.orchestra.run.trial195.stderr: 1 8 MallocExtension::Register 2026-02-24T16:09:26.481 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:26.481 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:09:26.500 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:09:26.500 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:09:26.548 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:09:26.548 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:09:26.555 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:26,554 seastar - Reactor backend: linux-aio 2026-02-24T16:09:26.563 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:26,562 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:26.563 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:26,562 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:26.567 INFO:teuthology.orchestra.run.trial195.stderr:==39700==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:09:26.567 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:26,566 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:09:26.731 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1195ac /lib64/libasan.so.8+0xfd239 /lib64/libasan.so.8+0xedcb5 0xe6ca27 0xe6cd08 0xa0d88c9 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.731 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xa0d87b8 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.731 INFO:teuthology.orchestra.run.trial195.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff90275c69 0xffffffff90420157 0xffffffff904203b0 0xffffffff9029525e 0xffffffff905e7269 0xffffffff905ca7d5 0xffffffff905d4bab 0xffffffff905d500b 0xffffffff902910d2 0xffffffff90f50451 0xffffffff91000ba2 2026-02-24T16:09:26.731 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15251 0x116816a 0xf741c5 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.763 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xa0ca1c4 0xa0cdce0 0xa0ce31e 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:26.825 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0x77b1595 0x77b1696 0x763e27c 0x7bc7238 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.825 INFO:teuthology.orchestra.run.trial195.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9041fee1 0xffffffff904203b0 0xffffffff9029525e 0xffffffff905e7269 0xffffffff905ca7d5 0xffffffff905d4bab 0xffffffff905d500b 0xffffffff902910d2 0xffffffff90f50451 0xffffffff91000ba2 2026-02-24T16:09:26.825 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xe55926 0x1168113 0xf741c5 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.825 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf0dbf /lib64/libasan.so.8+0xe908 /lib64/libasan.so.8+0xec776 0xcee0e1 0xa0d8868 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.851 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:26.908 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbf3 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.909 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297d5 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:26.909 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ce98e2 0x80977d2 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:27.011 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xe6c9de 0xe6cd08 0xa0d88c9 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x77b6764 0x76973b7 0x787c3c0 0x76e6d14 0x76e6e2d 0xa3a2446 0xa3a2649 0xa3a26e7 0xa3a276c 0xa3f5b6e 0xa3f5c27 0xa427569 0xa427910 0xa32dc97 0xa32dd50 0xa32dddf 0xa3f56d6 0xa3f5799 0xa3f58a6 0xa3f5a61 0x763eb45 0x7697625 0x769785c 0x7538bad 0x726516c 0x781cf48 0x781d09c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:27.049 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5af8 0x9fec307 0x9fec9e7 0xa0d221c 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:27.050 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe4b44 0x9fe500e 0x9fe5281 0x9fe6fa7 0xa0c5c7a 0xa0c7620 0xa0c7c7a 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:27.050 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191bff /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0x77b1595 0x77b1696 0x763e27c 0x7bc7238 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:27.132 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:27.134 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ce9c48 0x80977e7 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:27.135 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2c95a94 0x2cea88a 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:27.167 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea6d5 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:29.347 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:29.347 INFO:teuthology.orchestra.run.trial195.stderr:Suppressions used: 2026-02-24T16:09:29.348 INFO:teuthology.orchestra.run.trial195.stderr: count bytes template 2026-02-24T16:09:29.348 INFO:teuthology.orchestra.run.trial195.stderr: 1 8 MallocExtension::Register 2026-02-24T16:09:29.348 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:29.348 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:09:29.385 DEBUG:teuthology.orchestra.run.trial195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-24T16:09:29.410 DEBUG:teuthology.orchestra.run.trial195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:09:29.564 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:09:29.621 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:29,620 seastar - Reactor backend: linux-aio 2026-02-24T16:09:29.631 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:29,630 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:29.631 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:29,630 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:29.633 INFO:teuthology.orchestra.run.trial195.stderr:==39766==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:09:29.693 INFO:teuthology.orchestra.run.trial195.stderr:DEBUG 2026-02-24 16:09:29,692 [shard 0:main] seastar - reactor::drain 2026-02-24T16:09:31.797 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:31.797 INFO:teuthology.orchestra.run.trial195.stderr:Suppressions used: 2026-02-24T16:09:31.798 INFO:teuthology.orchestra.run.trial195.stderr: count bytes template 2026-02-24T16:09:31.798 INFO:teuthology.orchestra.run.trial195.stderr: 1 8 MallocExtension::Register 2026-02-24T16:09:31.798 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:31.798 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:09:31.817 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:09:31.817 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:09:31.859 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:09:31.859 INFO:teuthology.orchestra.run.trial195.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:09:31.866 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:31,865 seastar - Reactor backend: linux-aio 2026-02-24T16:09:31.876 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:31,875 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:31.876 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:31,875 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:09:31.880 INFO:teuthology.orchestra.run.trial195.stderr:==39764==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:09:31.880 INFO:teuthology.orchestra.run.trial195.stderr:INFO 2026-02-24 16:09:31,879 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:09:32.041 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9fe5454 0x9fe5af8 0x9fec307 0x9fec9e7 0xa0d221c 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.042 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0x9f2f2d6 0x9f2f3ec 0x9f2f4c8 0x9f2fb84 0x9f2fceb 0x9f300ef 0x9f303ac 0x9f311f4 0xa0a9b3b 0xa0a9c35 0x7eff112 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.042 INFO:teuthology.orchestra.run.trial195.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff905fa8b4 0xffffffff905fc7f9 0xffffffff905fdd21 0xffffffff90625a53 0xffffffff90625c15 0xffffffff905cc8ce 0xffffffff905d4bab 0xffffffff905d500b 0xffffffff902910d2 0xffffffff90f50451 0xffffffff91000ba2 2026-02-24T16:09:32.042 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129749 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.074 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x77b155f 0x77b1696 0x763e27c 0x7bc7238 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:32.104 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x9fe8592 0x9fe916d 0xa0ce105 0xa0d8c14 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.135 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b364 /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xba66a7 0x10d1d1e 0xf741d0 0x9fe4b44 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.137 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xcf9494 0x9f7ab9c 0xa0b490a 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.137 INFO:teuthology.orchestra.run.trial195.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9041ff44 0xffffffff904203b0 0xffffffff9029525e 0xffffffff905e7269 0xffffffff905ca7d5 0xffffffff905d4bab 0xffffffff905d500b 0xffffffff902910d2 0xffffffff90f50451 0xffffffff91000ba2 2026-02-24T16:09:32.160 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea83a 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:32.191 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.221 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x100697 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.221 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ebc /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.366 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9ff863f 0x9ff877b 0xa0b171c 0xa0b18b1 0xa0b1c43 0xa0b3229 0xa0b4544 0xa0b4964 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.366 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d9 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe6fa7 0x9fe74f4 0x9fe76f3 0xa0c7a65 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.366 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xa0c967e 0xa0cdce0 0xa0ce31e 0xa0d8c14 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.448 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:09:32.482 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ce97b9 0x80977d2 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.483 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:32.483 INFO:teuthology.orchestra.run.trial195.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1524e 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:09:34.704 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:34.704 INFO:teuthology.orchestra.run.trial195.stderr:Suppressions used: 2026-02-24T16:09:34.704 INFO:teuthology.orchestra.run.trial195.stderr: count bytes template 2026-02-24T16:09:34.704 INFO:teuthology.orchestra.run.trial195.stderr: 1 8 MallocExtension::Register 2026-02-24T16:09:34.705 INFO:teuthology.orchestra.run.trial195.stderr:----------------------------------------------------- 2026-02-24T16:09:34.705 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:09:34.742 DEBUG:teuthology.orchestra.run.trial195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-24T16:09:34.768 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-24T16:09:34.769 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:34.769 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-24T16:09:34.832 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:34.833 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-24T16:09:34.897 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:34.897 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-24T16:09:34.960 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:34.960 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-24T16:09:35.025 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:35.025 DEBUG:teuthology.orchestra.run.trial195:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-24T16:09:35.081 INFO:tasks.ceph:Adding keys to all mons... 2026-02-24T16:09:35.081 DEBUG:teuthology.orchestra.run.trial195:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-24T16:09:35.143 INFO:teuthology.orchestra.run.trial195.stdout:[mgr.x] 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout: key = AQAszZ1p1D97DRAA2hH0+qTOQnEiccGwKhOXgQ== 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout:[osd.0] 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout: key = AQAxzZ1pw1AVExAAeUkNsSutOOIHdDswsbbbKA== 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout:[osd.1] 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout: key = AQA2zZ1prPDYJRAA5UyfUlZqT/QrLv4zo6Wu1Q== 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout:[osd.2] 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout: key = AQA7zZ1ppg9sOBAAwf0QmPRdxH7//6Mp8BIhdw== 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout:[client.0] 2026-02-24T16:09:35.144 INFO:teuthology.orchestra.run.trial195.stdout: key = AQAszZ1p3icdEBAA2v0naqxCPsFr26KEXq7LbA== 2026-02-24T16:09:35.146 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-24T16:09:35.234 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:09:35.283 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:09:35.332 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:09:35.379 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-24T16:09:35.430 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-24T16:09:35.431 DEBUG:teuthology.orchestra.run.trial195:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-24T16:09:35.456 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-24T16:09:35.543 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.541+0000 7f31f1a87c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.543 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.541+0000 7f31f1a87c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.550 DEBUG:teuthology.orchestra.run.trial195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-24T16:09:35.575 DEBUG:teuthology.orchestra.run.trial195:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:09:35.632 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-24T16:09:35.633 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-24T16:09:35.633 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-24T16:09:35.675 INFO:tasks.ceph.mon.a:Started 2026-02-24T16:09:35.675 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-24T16:09:35.675 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-24T16:09:35.675 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-24T16:09:35.678 INFO:tasks.ceph.mgr.x:Started 2026-02-24T16:09:35.678 DEBUG:tasks.ceph:set 0 configs 2026-02-24T16:09:35.678 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph config dump 2026-02-24T16:09:35.737 INFO:tasks.ceph.mon.a.trial195.stderr:2026-02-24T16:09:35.735+0000 7fa11783ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.737 INFO:tasks.ceph.mon.a.trial195.stderr:2026-02-24T16:09:35.735+0000 7fa11783ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.737 INFO:tasks.ceph.mon.a.trial195.stderr:2026-02-24T16:09:35.735+0000 7fa11783ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.739 INFO:tasks.ceph.mgr.x.trial195.stderr:2026-02-24T16:09:35.737+0000 7fe39c1c9fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.755 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.752+0000 7fbc1ca3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.756 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.755+0000 7fbc1ca3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.788 INFO:teuthology.orchestra.run.trial195.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-24T16:09:35.795 INFO:tasks.ceph:Setting crush tunables to default 2026-02-24T16:09:35.795 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph osd crush tunables default 2026-02-24T16:09:35.870 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.865+0000 7faa0fa1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.871 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.869+0000 7faa0fa1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.904 INFO:teuthology.orchestra.run.trial195.stderr:adjusted tunables profile to default 2026-02-24T16:09:35.912 INFO:tasks.ceph:check_enable_crimson: True 2026-02-24T16:09:35.912 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-24T16:09:35.912 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-24T16:09:35.942 INFO:tasks.ceph.mgr.x.trial195.stderr:2026-02-24T16:09:35.940+0000 7fe39c1c9fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.942 INFO:tasks.ceph.mgr.x.trial195.stderr:2026-02-24T16:09:35.941+0000 7fe39c1c9fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.986 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.982+0000 7f839617e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:35.987 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:35.986+0000 7f839617e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:36.027 INFO:tasks.ceph.mon.a.trial195.stderr:2026-02-24T16:09:36.025+0000 7fa10ebf1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-24T16:09:36.027 INFO:tasks.ceph.mon.a.trial195.stderr:Please be aware that this feature is experimental, untested, 2026-02-24T16:09:36.027 INFO:tasks.ceph.mon.a.trial195.stderr:unsupported, and may result in data corruption, data loss, 2026-02-24T16:09:36.027 INFO:tasks.ceph.mon.a.trial195.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-24T16:09:36.027 INFO:tasks.ceph.mon.a.trial195.stderr:feature with important data. 2026-02-24T16:09:36.027 INFO:tasks.ceph.mon.a.trial195.stderr: 2026-02-24T16:09:36.763 INFO:tasks.ceph.mgr.x.trial195.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T16:09:36.763 INFO:tasks.ceph.mgr.x.trial195.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T16:09:36.763 INFO:tasks.ceph.mgr.x.trial195.stderr: from numpy import show_config as show_numpy_config 2026-02-24T16:09:36.914 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-24T16:09:36.915 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:36.915 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-24T16:09:36.980 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:36.980 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-24T16:09:37.045 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:37.045 DEBUG:teuthology.orchestra.run.trial195:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-24T16:09:37.109 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph osd new 01b58b4a-cd42-41fd-ab8a-e913982b1081 0 2026-02-24T16:09:37.223 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.218+0000 7ffa0a70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.224 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.222+0000 7ffa0a70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.260 INFO:teuthology.orchestra.run.trial195.stdout:0 2026-02-24T16:09:37.269 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph osd new baf66be6-09a9-4475-8c22-88a89fea4449 1 2026-02-24T16:09:37.342 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.337+0000 7f56c4a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.343 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.342+0000 7f56c4a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.379 INFO:teuthology.orchestra.run.trial195.stdout:1 2026-02-24T16:09:37.387 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph osd new ae62ea57-aea8-4758-8b77-e0cb9561668b 2 2026-02-24T16:09:37.461 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.457+0000 7f0442c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.462 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.461+0000 7f0442c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.499 INFO:teuthology.orchestra.run.trial195.stdout:2 2026-02-24T16:09:37.507 DEBUG:teuthology.orchestra.run.trial195:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-24T16:09:37.535 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-24T16:09:37.536 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-24T16:09:37.578 INFO:tasks.ceph.osd.0:Started 2026-02-24T16:09:37.579 DEBUG:teuthology.orchestra.run.trial195:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-24T16:09:37.607 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-24T16:09:37.608 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-24T16:09:37.649 INFO:tasks.ceph.osd.1:Started 2026-02-24T16:09:37.650 DEBUG:teuthology.orchestra.run.trial195:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-24T16:09:37.676 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-24T16:09:37.676 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-24T16:09:37.718 INFO:tasks.ceph.osd.2:Started 2026-02-24T16:09:37.718 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:09:37.796 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.793+0000 7f52f665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.798 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:37.796+0000 7f52f665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.831 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:37.831 INFO:teuthology.orchestra.run.trial195.stdout:{"epoch":6,"fsid":"8790c0b4-9560-4c5c-836a-a19ed753b087","created":"2026-02-24T16:09:35.747272+0000","modified":"2026-02-24T16:09:37.497567+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:09:37.497567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"01b58b4a-cd42-41fd-ab8a-e913982b1081","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"baf66be6-09a9-4475-8c22-88a89fea4449","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ae62ea57-aea8-4758-8b77-e0cb9561668b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:09:37.840 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-24T16:09:37.840 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-24T16:09:38.142 DEBUG:teuthology.orchestra.run.trial195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:09:38.209 INFO:teuthology.misc.health.trial195.stderr:2026-02-24T16:09:38.206+0000 7fe50a1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:38.211 INFO:teuthology.misc.health.trial195.stderr:2026-02-24T16:09:38.209+0000 7fe50a1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:38.244 INFO:teuthology.misc.health.trial195.stdout: 2026-02-24T16:09:38.244 INFO:teuthology.misc.health.trial195.stdout:{"epoch":6,"fsid":"8790c0b4-9560-4c5c-836a-a19ed753b087","created":"2026-02-24T16:09:35.747272+0000","modified":"2026-02-24T16:09:37.497567+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:09:37.497567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"01b58b4a-cd42-41fd-ab8a-e913982b1081","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"baf66be6-09a9-4475-8c22-88a89fea4449","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ae62ea57-aea8-4758-8b77-e0cb9561668b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:09:38.252 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:09:38.981 INFO:tasks.ceph.mgr.x.trial195.stderr:2026-02-24T16:09:38.979+0000 7fe2b917f640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-24T16:09:44.555 DEBUG:teuthology.orchestra.run.trial195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:09:44.640 INFO:teuthology.misc.health.trial195.stderr:2026-02-24T16:09:44.636+0000 7f5423c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:44.642 INFO:teuthology.misc.health.trial195.stderr:2026-02-24T16:09:44.640+0000 7f5423c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:44.782 INFO:teuthology.misc.health.trial195.stdout: 2026-02-24T16:09:44.782 INFO:teuthology.misc.health.trial195.stdout:{"epoch":7,"fsid":"8790c0b4-9560-4c5c-836a-a19ed753b087","created":"2026-02-24T16:09:35.747272+0000","modified":"2026-02-24T16:09:43.966165+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:09:37.497567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"01b58b4a-cd42-41fd-ab8a-e913982b1081","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"baf66be6-09a9-4475-8c22-88a89fea4449","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ae62ea57-aea8-4758-8b77-e0cb9561668b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:09:44.799 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:09:51.101 DEBUG:teuthology.orchestra.run.trial195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:09:51.171 INFO:teuthology.misc.health.trial195.stderr:2026-02-24T16:09:51.167+0000 7fa0ffd9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:51.172 INFO:teuthology.misc.health.trial195.stderr:2026-02-24T16:09:51.170+0000 7fa0ffd9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:51.283 INFO:teuthology.misc.health.trial195.stdout: 2026-02-24T16:09:51.283 INFO:teuthology.misc.health.trial195.stdout:{"epoch":13,"fsid":"8790c0b4-9560-4c5c-836a-a19ed753b087","created":"2026-02-24T16:09:35.747272+0000","modified":"2026-02-24T16:09:50.979758+0000","last_up_change":"2026-02-24T16:09:46.968049+0000","last_in_change":"2026-02-24T16:09:37.497567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:09:47.989397+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"01b58b4a-cd42-41fd-ab8a-e913982b1081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6803","nonce":921648518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6802","nonce":921648518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6812","nonce":921648518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6813","nonce":921648518}]},"public_addr":"10.20.193.195:6803/921648518","cluster_addr":"10.20.193.195:6802/921648518","heartbeat_back_addr":"10.20.193.195:6812/921648518","heartbeat_front_addr":"10.20.193.195:6813/921648518","state":["exists","up"]},{"osd":1,"uuid":"baf66be6-09a9-4475-8c22-88a89fea4449","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6808","nonce":1643076289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6807","nonce":1643076289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6810","nonce":1643076289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6811","nonce":1643076289}]},"public_addr":"10.20.193.195:6808/1643076289","cluster_addr":"10.20.193.195:6807/1643076289","heartbeat_back_addr":"10.20.193.195:6810/1643076289","heartbeat_front_addr":"10.20.193.195:6811/1643076289","state":["exists","up"]},{"osd":2,"uuid":"ae62ea57-aea8-4758-8b77-e0cb9561668b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6805","nonce":2775083266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6806","nonce":2775083266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6804","nonce":2775083266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6809","nonce":2775083266}]},"public_addr":"10.20.193.195:6805/2775083266","cluster_addr":"10.20.193.195:6806/2775083266","heartbeat_back_addr":"10.20.193.195:6804/2775083266","heartbeat_front_addr":"10.20.193.195:6809/2775083266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:09:51.291 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-24T16:09:51.291 INFO:tasks.ceph:Creating RBD pool 2026-02-24T16:09:51.291 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-24T16:09:51.366 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:51.360+0000 7f9055fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:51.368 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:51.365+0000 7f9055fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:51.986 INFO:teuthology.orchestra.run.trial195.stderr:pool 'rbd' created 2026-02-24T16:09:51.999 DEBUG:teuthology.orchestra.run.trial195:> rbd --cluster ceph pool init rbd 2026-02-24T16:09:52.041 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:52.038+0000 7fe3c5810240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:52.041 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:52.038+0000 7fe3c5810240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:52.043 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:52.041+0000 7fe3c5810240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.025 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-24T16:09:55.026 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-24T16:09:55.026 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-24T16:09:55.142 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.137+0000 7f3cf7b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.144 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.141+0000 7f3cf7b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.254 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:55.263 INFO:teuthology.orchestra.run.trial195.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-24T16:09:55.263 INFO:tasks.ceph_manager:config epoch is 1 2026-02-24T16:09:55.264 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-24T16:09:55.264 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-24T16:09:55.264 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-24T16:09:55.338 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.334+0000 7f973fdb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.339 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.337+0000 7f973fdb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.479 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:55.488 INFO:teuthology.orchestra.run.trial195.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6800","nonce":3897091278},{"type":"v1","addr":"10.20.193.195:6801","nonce":3897091278}]},"active_addr":"10.20.193.195:6801/3897091278","active_change":"2026-02-24T16:09:37.974731+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.195:0","nonce":32506447}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.195:0","nonce":3479568157}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.195:0","nonce":1456379260}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.195:0","nonce":1363786210}]}]} 2026-02-24T16:09:55.492 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-24T16:09:55.492 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-24T16:09:55.492 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:09:55.569 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.565+0000 7fcdc981a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.570 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.567+0000 7fcdc981a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.680 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:55.680 INFO:teuthology.orchestra.run.trial195.stdout:{"epoch":17,"fsid":"8790c0b4-9560-4c5c-836a-a19ed753b087","created":"2026-02-24T16:09:35.747272+0000","modified":"2026-02-24T16:09:54.994714+0000","last_up_change":"2026-02-24T16:09:46.968049+0000","last_in_change":"2026-02-24T16:09:37.497567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:09:47.989397+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:09:51.481011+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"01b58b4a-cd42-41fd-ab8a-e913982b1081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6803","nonce":921648518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6802","nonce":921648518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6812","nonce":921648518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6813","nonce":921648518}]},"public_addr":"10.20.193.195:6803/921648518","cluster_addr":"10.20.193.195:6802/921648518","heartbeat_back_addr":"10.20.193.195:6812/921648518","heartbeat_front_addr":"10.20.193.195:6813/921648518","state":["exists","up"]},{"osd":1,"uuid":"baf66be6-09a9-4475-8c22-88a89fea4449","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6808","nonce":1643076289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6807","nonce":1643076289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6810","nonce":1643076289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6811","nonce":1643076289}]},"public_addr":"10.20.193.195:6808/1643076289","cluster_addr":"10.20.193.195:6807/1643076289","heartbeat_back_addr":"10.20.193.195:6810/1643076289","heartbeat_front_addr":"10.20.193.195:6811/1643076289","state":["exists","up"]},{"osd":2,"uuid":"ae62ea57-aea8-4758-8b77-e0cb9561668b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6805","nonce":2775083266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6806","nonce":2775083266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6804","nonce":2775083266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6809","nonce":2775083266}]},"public_addr":"10.20.193.195:6805/2775083266","cluster_addr":"10.20.193.195:6806/2775083266","heartbeat_back_addr":"10.20.193.195:6804/2775083266","heartbeat_front_addr":"10.20.193.195:6809/2775083266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:09:55.689 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-24T16:09:55.690 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:09:55.766 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.762+0000 7f6fbcecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.767 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.765+0000 7f6fbcecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.878 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:55.878 INFO:teuthology.orchestra.run.trial195.stdout:{"epoch":17,"fsid":"8790c0b4-9560-4c5c-836a-a19ed753b087","created":"2026-02-24T16:09:35.747272+0000","modified":"2026-02-24T16:09:54.994714+0000","last_up_change":"2026-02-24T16:09:46.968049+0000","last_in_change":"2026-02-24T16:09:37.497567+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:09:47.989397+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:09:51.481011+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"01b58b4a-cd42-41fd-ab8a-e913982b1081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6803","nonce":921648518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6802","nonce":921648518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6812","nonce":921648518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6813","nonce":921648518}]},"public_addr":"10.20.193.195:6803/921648518","cluster_addr":"10.20.193.195:6802/921648518","heartbeat_back_addr":"10.20.193.195:6812/921648518","heartbeat_front_addr":"10.20.193.195:6813/921648518","state":["exists","up"]},{"osd":1,"uuid":"baf66be6-09a9-4475-8c22-88a89fea4449","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6808","nonce":1643076289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6807","nonce":1643076289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6810","nonce":1643076289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6811","nonce":1643076289}]},"public_addr":"10.20.193.195:6808/1643076289","cluster_addr":"10.20.193.195:6807/1643076289","heartbeat_back_addr":"10.20.193.195:6810/1643076289","heartbeat_front_addr":"10.20.193.195:6811/1643076289","state":["exists","up"]},{"osd":2,"uuid":"ae62ea57-aea8-4758-8b77-e0cb9561668b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6805","nonce":2775083266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6806","nonce":2775083266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6804","nonce":2775083266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.195:6809","nonce":2775083266}]},"public_addr":"10.20.193.195:6805/2775083266","cluster_addr":"10.20.193.195:6806/2775083266","heartbeat_back_addr":"10.20.193.195:6804/2775083266","heartbeat_front_addr":"10.20.193.195:6809/2775083266","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:09:55.888 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-24T16:09:55.889 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-24T16:09:55.889 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-24T16:09:55.968 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.964+0000 7f218f30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.969 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.964+0000 7f76d0a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.970 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.967+0000 7f218f30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.970 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.965+0000 7f9c1185a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.970 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.968+0000 7f76d0a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.971 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:55.969+0000 7f9c1185a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:55.983 INFO:teuthology.orchestra.run.trial195.stdout:42949672968 2026-02-24T16:09:55.984 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T16:09:55.985 INFO:teuthology.orchestra.run.trial195.stdout:42949672968 2026-02-24T16:09:55.986 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T16:09:55.987 INFO:teuthology.orchestra.run.trial195.stdout:38654705673 2026-02-24T16:09:55.987 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T16:09:56.065 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.060+0000 7fe3ed19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.066 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.064+0000 7fe3ed19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.106 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.101+0000 7f8120e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.107 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.102+0000 7f65aef7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.108 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.105+0000 7f8120e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.108 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.106+0000 7f65aef7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.179 INFO:teuthology.orchestra.run.trial195.stdout:42949672968 2026-02-24T16:09:56.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.1 2026-02-24T16:09:56.191 DEBUG:teuthology.parallel:result is None 2026-02-24T16:09:56.224 INFO:teuthology.orchestra.run.trial195.stdout:42949672968 2026-02-24T16:09:56.224 INFO:teuthology.orchestra.run.trial195.stdout:38654705673 2026-02-24T16:09:56.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.2 2026-02-24T16:09:56.235 DEBUG:teuthology.parallel:result is None 2026-02-24T16:09:56.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.0 2026-02-24T16:09:56.236 DEBUG:teuthology.parallel:result is None 2026-02-24T16:09:56.236 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T16:09:56.236 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:09:56.352 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.348+0000 7f3305174640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.353 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.351+0000 7f3305174640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.461 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:56.462 INFO:teuthology.orchestra.run.trial195.stderr:dumped all 2026-02-24T16:09:56.470 INFO:teuthology.orchestra.run.trial195.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-24T16:09:55.978865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2492,"kb_used_data":2492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497476,"statfs":{"total":160255967232,"available":160253415424,"internally_reserved":0,"allocated":2551808,"data_stored":2551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.994689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.009336+0000","last_change":"2026-02-24T16:09:55.013719+0000","last_active":"2026-02-24T16:09:55.009336+0000","last_peered":"2026-02-24T16:09:55.009336+0000","last_clean":"2026-02-24T16:09:55.009336+0000","last_became_active":"2026-02-24T16:09:53.028544+0000","last_became_peered":"2026-02-24T16:09:53.028544+0000","last_unstale":"2026-02-24T16:09:55.009336+0000","last_undegraded":"2026-02-24T16:09:55.009336+0000","last_fullsized":"2026-02-24T16:09:55.009336+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.010351+0000","last_change":"2026-02-24T16:09:55.013751+0000","last_active":"2026-02-24T16:09:55.010351+0000","last_peered":"2026-02-24T16:09:55.010351+0000","last_clean":"2026-02-24T16:09:55.010351+0000","last_became_active":"2026-02-24T16:09:53.025947+0000","last_became_peered":"2026-02-24T16:09:53.025947+0000","last_unstale":"2026-02-24T16:09:55.010351+0000","last_undegraded":"2026-02-24T16:09:55.010351+0000","last_fullsized":"2026-02-24T16:09:55.010351+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.010778+0000","last_change":"2026-02-24T16:09:55.015952+0000","last_active":"2026-02-24T16:09:55.010778+0000","last_peered":"2026-02-24T16:09:55.010778+0000","last_clean":"2026-02-24T16:09:55.010778+0000","last_became_active":"2026-02-24T16:09:53.030220+0000","last_became_peered":"2026-02-24T16:09:53.030220+0000","last_unstale":"2026-02-24T16:09:55.010778+0000","last_undegraded":"2026-02-24T16:09:55.010778+0000","last_fullsized":"2026-02-24T16:09:55.010778+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.011304+0000","last_change":"2026-02-24T16:09:55.015422+0000","last_active":"2026-02-24T16:09:55.011304+0000","last_peered":"2026-02-24T16:09:55.011304+0000","last_clean":"2026-02-24T16:09:55.011304+0000","last_became_active":"2026-02-24T16:09:53.032436+0000","last_became_peered":"2026-02-24T16:09:53.032436+0000","last_unstale":"2026-02-24T16:09:55.011304+0000","last_undegraded":"2026-02-24T16:09:55.011304+0000","last_fullsized":"2026-02-24T16:09:55.011304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.011880+0000","last_change":"2026-02-24T16:09:55.011880+0000","last_active":"2026-02-24T16:09:55.011880+0000","last_peered":"2026-02-24T16:09:55.011880+0000","last_clean":"2026-02-24T16:09:55.011880+0000","last_became_active":"2026-02-24T16:09:53.035710+0000","last_became_peered":"2026-02-24T16:09:53.035710+0000","last_unstale":"2026-02-24T16:09:55.011880+0000","last_undegraded":"2026-02-24T16:09:55.011880+0000","last_fullsized":"2026-02-24T16:09:55.011880+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.008886+0000","last_change":"2026-02-24T16:09:55.011302+0000","last_active":"2026-02-24T16:09:55.008886+0000","last_peered":"2026-02-24T16:09:55.008886+0000","last_clean":"2026-02-24T16:09:55.008886+0000","last_became_active":"2026-02-24T16:09:53.029898+0000","last_became_peered":"2026-02-24T16:09:53.029898+0000","last_unstale":"2026-02-24T16:09:55.008886+0000","last_undegraded":"2026-02-24T16:09:55.008886+0000","last_fullsized":"2026-02-24T16:09:55.008886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.006565+0000","last_change":"2026-02-24T16:09:55.009666+0000","last_active":"2026-02-24T16:09:55.006565+0000","last_peered":"2026-02-24T16:09:55.006565+0000","last_clean":"2026-02-24T16:09:55.006565+0000","last_became_active":"2026-02-24T16:09:53.031805+0000","last_became_peered":"2026-02-24T16:09:53.031805+0000","last_unstale":"2026-02-24T16:09:55.006565+0000","last_undegraded":"2026-02-24T16:09:55.006565+0000","last_fullsized":"2026-02-24T16:09:55.006565+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.011353+0000","last_change":"2026-02-24T16:09:55.015945+0000","last_active":"2026-02-24T16:09:55.011353+0000","last_peered":"2026-02-24T16:09:55.011353+0000","last_clean":"2026-02-24T16:09:55.011353+0000","last_became_active":"2026-02-24T16:09:53.025201+0000","last_became_peered":"2026-02-24T16:09:53.025201+0000","last_unstale":"2026-02-24T16:09:55.011353+0000","last_undegraded":"2026-02-24T16:09:55.011353+0000","last_fullsized":"2026-02-24T16:09:55.011353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.009233+0000","last_change":"2026-02-24T16:09:50.009985+0000","last_active":"2026-02-24T16:09:55.009233+0000","last_peered":"2026-02-24T16:09:55.009233+0000","last_clean":"2026-02-24T16:09:55.009233+0000","last_became_active":"2026-02-24T16:09:50.002835+0000","last_became_peered":"2026-02-24T16:09:50.002835+0000","last_unstale":"2026-02-24T16:09:55.009233+0000","last_undegraded":"2026-02-24T16:09:55.009233+0000","last_fullsized":"2026-02-24T16:09:55.009233+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:48.971670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:48.971670+0000","last_clean_scrub_stamp":"2026-02-24T16:09:48.971670+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1116,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165540,"statfs":{"total":53418655744,"available":53417512960,"internally_reserved":0,"allocated":1142784,"data_stored":1142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:09:56.472 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:09:56.547 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.543+0000 7fd53d2c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.548 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.546+0000 7fd53d2c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.659 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:56.659 INFO:teuthology.orchestra.run.trial195.stderr:dumped all 2026-02-24T16:09:56.668 INFO:teuthology.orchestra.run.trial195.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-24T16:09:55.978865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2492,"kb_used_data":2492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497476,"statfs":{"total":160255967232,"available":160253415424,"internally_reserved":0,"allocated":2551808,"data_stored":2551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.994689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.009336+0000","last_change":"2026-02-24T16:09:55.013719+0000","last_active":"2026-02-24T16:09:55.009336+0000","last_peered":"2026-02-24T16:09:55.009336+0000","last_clean":"2026-02-24T16:09:55.009336+0000","last_became_active":"2026-02-24T16:09:53.028544+0000","last_became_peered":"2026-02-24T16:09:53.028544+0000","last_unstale":"2026-02-24T16:09:55.009336+0000","last_undegraded":"2026-02-24T16:09:55.009336+0000","last_fullsized":"2026-02-24T16:09:55.009336+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.010351+0000","last_change":"2026-02-24T16:09:55.013751+0000","last_active":"2026-02-24T16:09:55.010351+0000","last_peered":"2026-02-24T16:09:55.010351+0000","last_clean":"2026-02-24T16:09:55.010351+0000","last_became_active":"2026-02-24T16:09:53.025947+0000","last_became_peered":"2026-02-24T16:09:53.025947+0000","last_unstale":"2026-02-24T16:09:55.010351+0000","last_undegraded":"2026-02-24T16:09:55.010351+0000","last_fullsized":"2026-02-24T16:09:55.010351+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.010778+0000","last_change":"2026-02-24T16:09:55.015952+0000","last_active":"2026-02-24T16:09:55.010778+0000","last_peered":"2026-02-24T16:09:55.010778+0000","last_clean":"2026-02-24T16:09:55.010778+0000","last_became_active":"2026-02-24T16:09:53.030220+0000","last_became_peered":"2026-02-24T16:09:53.030220+0000","last_unstale":"2026-02-24T16:09:55.010778+0000","last_undegraded":"2026-02-24T16:09:55.010778+0000","last_fullsized":"2026-02-24T16:09:55.010778+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.011304+0000","last_change":"2026-02-24T16:09:55.015422+0000","last_active":"2026-02-24T16:09:55.011304+0000","last_peered":"2026-02-24T16:09:55.011304+0000","last_clean":"2026-02-24T16:09:55.011304+0000","last_became_active":"2026-02-24T16:09:53.032436+0000","last_became_peered":"2026-02-24T16:09:53.032436+0000","last_unstale":"2026-02-24T16:09:55.011304+0000","last_undegraded":"2026-02-24T16:09:55.011304+0000","last_fullsized":"2026-02-24T16:09:55.011304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.011880+0000","last_change":"2026-02-24T16:09:55.011880+0000","last_active":"2026-02-24T16:09:55.011880+0000","last_peered":"2026-02-24T16:09:55.011880+0000","last_clean":"2026-02-24T16:09:55.011880+0000","last_became_active":"2026-02-24T16:09:53.035710+0000","last_became_peered":"2026-02-24T16:09:53.035710+0000","last_unstale":"2026-02-24T16:09:55.011880+0000","last_undegraded":"2026-02-24T16:09:55.011880+0000","last_fullsized":"2026-02-24T16:09:55.011880+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.008886+0000","last_change":"2026-02-24T16:09:55.011302+0000","last_active":"2026-02-24T16:09:55.008886+0000","last_peered":"2026-02-24T16:09:55.008886+0000","last_clean":"2026-02-24T16:09:55.008886+0000","last_became_active":"2026-02-24T16:09:53.029898+0000","last_became_peered":"2026-02-24T16:09:53.029898+0000","last_unstale":"2026-02-24T16:09:55.008886+0000","last_undegraded":"2026-02-24T16:09:55.008886+0000","last_fullsized":"2026-02-24T16:09:55.008886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.006565+0000","last_change":"2026-02-24T16:09:55.009666+0000","last_active":"2026-02-24T16:09:55.006565+0000","last_peered":"2026-02-24T16:09:55.006565+0000","last_clean":"2026-02-24T16:09:55.006565+0000","last_became_active":"2026-02-24T16:09:53.031805+0000","last_became_peered":"2026-02-24T16:09:53.031805+0000","last_unstale":"2026-02-24T16:09:55.006565+0000","last_undegraded":"2026-02-24T16:09:55.006565+0000","last_fullsized":"2026-02-24T16:09:55.006565+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.011353+0000","last_change":"2026-02-24T16:09:55.015945+0000","last_active":"2026-02-24T16:09:55.011353+0000","last_peered":"2026-02-24T16:09:55.011353+0000","last_clean":"2026-02-24T16:09:55.011353+0000","last_became_active":"2026-02-24T16:09:53.025201+0000","last_became_peered":"2026-02-24T16:09:53.025201+0000","last_unstale":"2026-02-24T16:09:55.011353+0000","last_undegraded":"2026-02-24T16:09:55.011353+0000","last_fullsized":"2026-02-24T16:09:55.011353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:51.983218+0000","last_clean_scrub_stamp":"2026-02-24T16:09:51.983218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:09:55.009233+0000","last_change":"2026-02-24T16:09:50.009985+0000","last_active":"2026-02-24T16:09:55.009233+0000","last_peered":"2026-02-24T16:09:55.009233+0000","last_clean":"2026-02-24T16:09:55.009233+0000","last_became_active":"2026-02-24T16:09:50.002835+0000","last_became_peered":"2026-02-24T16:09:50.002835+0000","last_unstale":"2026-02-24T16:09:55.009233+0000","last_undegraded":"2026-02-24T16:09:55.009233+0000","last_fullsized":"2026-02-24T16:09:55.009233+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:09:48.971670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:09:48.971670+0000","last_clean_scrub_stamp":"2026-02-24T16:09:48.971670+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1116,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165540,"statfs":{"total":53418655744,"available":53417512960,"internally_reserved":0,"allocated":1142784,"data_stored":1142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:09:56.669 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T16:09:56.670 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-24T16:09:56.670 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-24T16:09:56.670 DEBUG:teuthology.orchestra.run.trial195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-24T16:09:56.747 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.743+0000 7fd600a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.748 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:09:56.746+0000 7fd600a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:56.886 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:09:56.886 INFO:teuthology.orchestra.run.trial195.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-24T16:09:56.895 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-24T16:09:56.896 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-24T16:09:57.023 DEBUG:teuthology.orchestra.run.trial195:> echo 'StrictHostKeyChecking no 2026-02-24T16:09:57.023 DEBUG:teuthology.orchestra.run.trial195:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-24T16:09:57.039 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQDNeyHLdFtQGtdQQLb6aAztN0xbdirlRoJHmmIB8zi9kPa9c/99Vuswyaiyy+hsh+mQzzBk/5QCRl3frCwXjhc0ERgB/Buwm0I1YUThsco04sowrLOgatpvlyGIZ141b+jl2uvyooL/98qMefHZJWXvtI6wjzNK7iPF27ZvwFDDhbYdtEEkeTPLfZ2UUWVVTUmsio2OpFoVLdlSOSEGnF4Icno90lfSx/yljlXFJpvrNNpi9o/Dt2BHxTJefiJfkcSo7fTZCvf1KikdQi5k0giGGLb+l0b+GyGjVBFVh5piyAHdf8Wf5XUa+0eMv2VfHv5EKuUMaodoyHzNOH0z7PRH 2026-02-24T16:09:57.039 INFO:teuthology.task.ssh_keys:pushing keys to trial195.front.sepia.ceph.com for ubuntu 2026-02-24T16:09:57.039 DEBUG:teuthology.orchestra.run.trial195:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:09:57.095 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:57.095 DEBUG:teuthology.orchestra.run.trial195:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-24T16:09:57.095 DEBUG:teuthology.orchestra.run.trial195:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-24T16:09:57.154 DEBUG:teuthology.orchestra.run.trial195:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:09:57.210 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:57.210 DEBUG:teuthology.orchestra.run.trial195:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:09:57.266 DEBUG:teuthology.orchestra.run.trial195:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:09:57:265981 2026-02-24T16:09:57.322 INFO:teuthology.orchestra.run.trial195.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-24_16:09:57:265981' 2026-02-24T16:09:57.322 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:57.322 DEBUG:teuthology.orchestra.run.trial195:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-24T16:09:57.378 INFO:teuthology.run_tasks:Running task cbt... 2026-02-24T16:09:57.402 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial195.front.sepia.ceph.com', 'osds': ['ubuntu@trial195.front.sepia.ceph.com'], 'mons': ['ubuntu@trial195.front.sepia.ceph.com'], 'clients': ['ubuntu@trial195.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial195.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-24T16:09:57.403 DEBUG:teuthology.orchestra.run.trial195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-24T16:09:57.437 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:09:57.438 DEBUG:teuthology.orchestra.run.trial195:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:09:57.490 DEBUG:teuthology.orchestra.run.trial195:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-24T16:09:57.553 INFO:teuthology.orchestra.run.trial195.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-24T16:09:57.808 INFO:tasks.cbt:Installing collectl 2026-02-24T16:09:57.808 DEBUG:teuthology.orchestra.run.trial195:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-24T16:09:57.883 INFO:teuthology.orchestra.run.trial195.stderr:--2026-02-24 16:09:57-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-24T16:09:57.939 INFO:teuthology.orchestra.run.trial195.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:c95, ... 2026-02-24T16:09:57.942 INFO:teuthology.orchestra.run.trial195.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-24T16:09:58.145 INFO:teuthology.orchestra.run.trial195.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:09:58.146 INFO:teuthology.orchestra.run.trial195.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnc1W8RG-QmxiiFcAhtXvt6_QGVM0cLsXcKmD2ckW5dwfiWYrUd9coN1RA1R6YhioE7tfwTDjq5ArNJMIHJKGpibxVg%3D%3D&use_mirror=phoenixnap&r= [following] 2026-02-24T16:09:58.146 INFO:teuthology.orchestra.run.trial195.stderr:--2026-02-24 16:09:58-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnc1W8RG-QmxiiFcAhtXvt6_QGVM0cLsXcKmD2ckW5dwfiWYrUd9coN1RA1R6YhioE7tfwTDjq5ArNJMIHJKGpibxVg%3D%3D&use_mirror=phoenixnap&r= 2026-02-24T16:09:58.194 INFO:teuthology.orchestra.run.trial195.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-24T16:09:58.197 INFO:teuthology.orchestra.run.trial195.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.13.149|:443... connected. 2026-02-24T16:09:58.382 INFO:teuthology.orchestra.run.trial195.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:09:58.382 INFO:teuthology.orchestra.run.trial195.stderr:Location: https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-24T16:09:58.383 INFO:teuthology.orchestra.run.trial195.stderr:--2026-02-24 16:09:58-- https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-24T16:09:58.388 INFO:teuthology.orchestra.run.trial195.stderr:Resolving phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)... 184.164.141.26 2026-02-24T16:09:58.451 INFO:teuthology.orchestra.run.trial195.stderr:Connecting to phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)|184.164.141.26|:443... connected. 2026-02-24T16:09:59.206 INFO:teuthology.orchestra.run.trial195.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-24T16:09:59.206 INFO:teuthology.orchestra.run.trial195.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-24T16:09:59.206 INFO:teuthology.orchestra.run.trial195.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-24T16:09:59.206 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:09:59.268 INFO:teuthology.orchestra.run.trial195.stderr: 0K .......... .......... .......... .......... .......... 8% 794K 1s 2026-02-24T16:09:59.332 INFO:teuthology.orchestra.run.trial195.stderr: 50K .......... .......... .......... .......... .......... 16% 781K 1s 2026-02-24T16:09:59.341 INFO:teuthology.orchestra.run.trial195.stderr: 100K .......... .......... .......... .......... .......... 24% 5.96M 0s 2026-02-24T16:09:59.401 INFO:teuthology.orchestra.run.trial195.stderr: 150K .......... .......... .......... .......... .......... 32% 831K 0s 2026-02-24T16:09:59.406 INFO:teuthology.orchestra.run.trial195.stderr: 200K .......... .......... .......... .......... .......... 41% 13.6M 0s 2026-02-24T16:09:59.406 INFO:teuthology.orchestra.run.trial195.stderr: 250K .......... .......... .......... .......... .......... 49% 403M 0s 2026-02-24T16:09:59.469 INFO:teuthology.orchestra.run.trial195.stderr: 300K .......... .......... .......... .......... .......... 57% 769K 0s 2026-02-24T16:09:59.472 INFO:teuthology.orchestra.run.trial195.stderr: 350K .......... .......... .......... .......... .......... 65% 22.5M 0s 2026-02-24T16:09:59.592 INFO:teuthology.orchestra.run.trial195.stderr: 400K .......... .......... .......... .......... .......... 74% 421K 0s 2026-02-24T16:09:59.593 INFO:teuthology.orchestra.run.trial195.stderr: 450K .......... .......... .......... .......... .......... 82% 198M 0s 2026-02-24T16:10:00.457 INFO:teuthology.orchestra.run.trial195.stderr: 500K .......... .......... .......... .......... .......... 90% 57.7K 0s 2026-02-24T16:10:00.520 INFO:teuthology.orchestra.run.trial195.stderr: 550K .......... .......... .......... .......... .......... 98% 800K 0s 2026-02-24T16:10:00.520 INFO:teuthology.orchestra.run.trial195.stderr: 600K ...... 100% 11.4T=1.3s 2026-02-24T16:10:00.520 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:10:00.520 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24 16:10:00 (461 KB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-24T16:10:00.520 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:10:00.552 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/ 2026-02-24T16:10:00.552 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/collectl.conf 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/ 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Socket.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/colmux.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Import.html 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/style.css 2026-02-24T16:10:00.553 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Process.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-24T16:10:00.554 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Export.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Startup.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Performance.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Messages.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Logging.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-24T16:10:00.555 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Misc.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Examples.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Data.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-24T16:10:00.556 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Playback.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Hello.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Network.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Memory.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/Disks.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/service/ 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/service/collectl.service 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/initd/ 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-24T16:10:00.557 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/initd/collectl 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/misc.ph 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/gexpr.ph 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/statsd.ph 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/envrules.std 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/hello.ph 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/vmsum.ph 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/proctree.ph 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/README-WINDOWS 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/formatit.ph 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/colmux 2026-02-24T16:10:00.558 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/INSTALL 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/graphite.ph 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/vmstat.ph 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/README 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/man1/ 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/man1/collectl.1 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/man1/colmux.1 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/lexpr.ph 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/client.pl 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/collectl 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/COPYING 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/vnet.ph 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/GPL 2026-02-24T16:10:00.559 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/UNINSTALL 2026-02-24T16:10:00.560 INFO:teuthology.orchestra.run.trial195.stdout:collectl-4.3.1/ARTISTIC 2026-02-24T16:10:00.594 INFO:teuthology.orchestra.run.trial195.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-24T16:10:00.596 DEBUG:teuthology.orchestra.run.trial195:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-24T16:10:00.958 INFO:teuthology.orchestra.run.trial195.stdout:Last metadata expiration check: 0:00:59 ago on Tue 24 Feb 2026 04:09:01 PM UTC. 2026-02-24T16:10:01.049 INFO:teuthology.orchestra.run.trial195.stdout:Dependencies resolved. 2026-02-24T16:10:01.049 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout: Package Arch Version Repository Size 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout:Installing: 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout: librbd-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 24 k 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout:Installing dependencies: 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout: libradospp-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 31 k 2026-02-24T16:10:01.050 INFO:teuthology.orchestra.run.trial195.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout:Transaction Summary 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout:================================================================================ 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout:Install 6 Packages 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout:Total download size: 3.8 M 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout:Installed size: 14 M 2026-02-24T16:10:01.051 INFO:teuthology.orchestra.run.trial195.stdout:Downloading Packages: 2026-02-24T16:10:01.330 INFO:teuthology.orchestra.run.trial195.stdout:(1/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 3.3 MB/s | 290 kB 00:00 2026-02-24T16:10:01.366 INFO:teuthology.orchestra.run.trial195.stdout:(2/6): perf-5.14.0-686.el9.x86_64.rpm 91 MB/s | 3.3 MB 00:00 2026-02-24T16:10:01.595 INFO:teuthology.orchestra.run.trial195.stdout:(3/6): librbd-devel-20.3.0-5343.gd26e5c22.el9.x 68 kB/s | 24 kB 00:00 2026-02-24T16:10:01.625 INFO:teuthology.orchestra.run.trial195.stdout:(4/6): pdsh-2.34-7.el9.x86_64.rpm 457 kB/s | 118 kB 00:00 2026-02-24T16:10:01.669 INFO:teuthology.orchestra.run.trial195.stdout:(5/6): libradospp-devel-20.3.0-5343.gd26e5c22.e 73 kB/s | 31 kB 00:00 2026-02-24T16:10:01.736 INFO:teuthology.orchestra.run.trial195.stdout:(6/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 93 kB/s | 13 kB 00:00 2026-02-24T16:10:01.738 INFO:teuthology.orchestra.run.trial195.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:10:01.739 INFO:teuthology.orchestra.run.trial195.stdout:Total 5.5 MB/s | 3.8 MB 00:00 2026-02-24T16:10:01.763 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction check 2026-02-24T16:10:01.774 INFO:teuthology.orchestra.run.trial195.stdout:Transaction check succeeded. 2026-02-24T16:10:01.774 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction test 2026-02-24T16:10:01.831 INFO:teuthology.orchestra.run.trial195.stdout:Transaction test succeeded. 2026-02-24T16:10:01.831 INFO:teuthology.orchestra.run.trial195.stdout:Running transaction 2026-02-24T16:10:01.906 INFO:teuthology.orchestra.run.trial195.stdout: Preparing : 1/1 2026-02-24T16:10:01.912 INFO:teuthology.orchestra.run.trial195.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-24T16:10:01.953 INFO:teuthology.orchestra.run.trial195.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-24T16:10:01.957 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:10:01.960 INFO:teuthology.orchestra.run.trial195.stdout: Installing : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/6 2026-02-24T16:10:02.125 INFO:teuthology.orchestra.run.trial195.stdout: Installing : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/6 2026-02-24T16:10:02.147 INFO:teuthology.orchestra.run.trial195.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:10:02.263 INFO:teuthology.orchestra.run.trial195.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:10:02.264 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/6 2026-02-24T16:10:02.264 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/6 2026-02-24T16:10:02.264 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:10:02.264 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-24T16:10:02.264 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-24T16:10:02.328 INFO:teuthology.orchestra.run.trial195.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout:Installed: 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: pdsh-2.34-7.el9.x86_64 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: perf-5.14.0-686.el9.x86_64 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:10:02.329 INFO:teuthology.orchestra.run.trial195.stdout:Complete! 2026-02-24T16:10:02.359 DEBUG:teuthology.orchestra.run.trial195:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-24T16:10:02.499 INFO:teuthology.orchestra.run.trial195.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-24T16:10:02.509 INFO:teuthology.orchestra.run.trial195.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-24T16:10:02.972 INFO:teuthology.orchestra.run.trial195.stdout:Collecting lxml 2026-02-24T16:10:03.005 INFO:teuthology.orchestra.run.trial195.stdout: Downloading lxml-6.0.2-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.3 MB) 2026-02-24T16:10:03.371 INFO:teuthology.orchestra.run.trial195.stdout:Collecting matplotlib 2026-02-24T16:10:03.377 INFO:teuthology.orchestra.run.trial195.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-24T16:10:03.445 INFO:teuthology.orchestra.run.trial195.stdout:Collecting cycler>=0.10 2026-02-24T16:10:03.452 INFO:teuthology.orchestra.run.trial195.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-24T16:10:03.453 INFO:teuthology.orchestra.run.trial195.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-24T16:10:03.784 INFO:teuthology.orchestra.run.trial195.stdout:Collecting fonttools>=4.22.0 2026-02-24T16:10:03.790 INFO:teuthology.orchestra.run.trial195.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-24T16:10:03.967 INFO:teuthology.orchestra.run.trial195.stdout:Collecting contourpy>=1.0.1 2026-02-24T16:10:03.979 INFO:teuthology.orchestra.run.trial195.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-24T16:10:04.018 INFO:teuthology.orchestra.run.trial195.stdout:Collecting importlib-resources>=3.2.0 2026-02-24T16:10:04.022 INFO:teuthology.orchestra.run.trial195.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-24T16:10:04.024 INFO:teuthology.orchestra.run.trial195.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-24T16:10:04.187 INFO:teuthology.orchestra.run.trial195.stdout:Collecting kiwisolver>=1.3.1 2026-02-24T16:10:04.192 INFO:teuthology.orchestra.run.trial195.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-24T16:10:04.202 INFO:teuthology.orchestra.run.trial195.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-24T16:10:04.203 INFO:teuthology.orchestra.run.trial195.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-24T16:10:04.582 INFO:teuthology.orchestra.run.trial195.stdout:Collecting pillow>=8 2026-02-24T16:10:04.587 INFO:teuthology.orchestra.run.trial195.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-24T16:10:04.706 INFO:teuthology.orchestra.run.trial195.stdout:Collecting zipp>=3.1.0 2026-02-24T16:10:04.709 INFO:teuthology.orchestra.run.trial195.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-24T16:10:04.720 INFO:teuthology.orchestra.run.trial195.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-24T16:10:04.833 INFO:teuthology.orchestra.run.trial195.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib, lxml 2026-02-24T16:10:05.964 INFO:teuthology.orchestra.run.trial195.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 lxml-6.0.2 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-24T16:10:06.088 INFO:tasks.cbt:benchmark: librbdfio 2026-02-24T16:10:06.088 DEBUG:teuthology.orchestra.run.trial195:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2026-02-24T16:10:06.106 INFO:teuthology.orchestra.run.trial195.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2026-02-24T16:10:06.903 DEBUG:teuthology.orchestra.run.trial195:> cd /home/ubuntu/cephtest/fio && ./configure && make 2026-02-24T16:10:07.074 INFO:teuthology.orchestra.run.trial195.stdout:Operating system Linux 2026-02-24T16:10:07.075 INFO:teuthology.orchestra.run.trial195.stdout:CPU x86_64 2026-02-24T16:10:07.075 INFO:teuthology.orchestra.run.trial195.stdout:Big endian no 2026-02-24T16:10:07.075 INFO:teuthology.orchestra.run.trial195.stdout:Compiler gcc 2026-02-24T16:10:07.075 INFO:teuthology.orchestra.run.trial195.stdout:Cross compile no 2026-02-24T16:10:07.075 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:10:07.075 INFO:teuthology.orchestra.run.trial195.stdout:Static build no 2026-02-24T16:10:07.121 INFO:teuthology.orchestra.run.trial195.stdout:Wordsize 64 2026-02-24T16:10:07.127 INFO:teuthology.orchestra.run.trial195.stdout:zlib no 2026-02-24T16:10:07.138 INFO:teuthology.orchestra.run.trial195.stdout:fcntl(F_FULLFSYNC) no 2026-02-24T16:10:07.148 INFO:teuthology.orchestra.run.trial195.stdout:Linux AIO support no 2026-02-24T16:10:07.149 INFO:teuthology.orchestra.run.trial195.stdout:Linux AIO support rw flags no 2026-02-24T16:10:07.172 INFO:teuthology.orchestra.run.trial195.stdout:POSIX AIO support yes 2026-02-24T16:10:07.172 INFO:teuthology.orchestra.run.trial195.stdout:POSIX AIO support needs -lrt no 2026-02-24T16:10:07.192 INFO:teuthology.orchestra.run.trial195.stdout:POSIX AIO fsync yes 2026-02-24T16:10:07.214 INFO:teuthology.orchestra.run.trial195.stdout:POSIX pshared support yes 2026-02-24T16:10:07.241 INFO:teuthology.orchestra.run.trial195.stdout:pthread_condattr_setclock() yes 2026-02-24T16:10:07.265 INFO:teuthology.orchestra.run.trial195.stdout:pthread_sigmask() yes 2026-02-24T16:10:07.290 INFO:teuthology.orchestra.run.trial195.stdout:pthread_getaffinity_np() yes 2026-02-24T16:10:07.297 INFO:teuthology.orchestra.run.trial195.stdout:Solaris AIO support no 2026-02-24T16:10:07.318 INFO:teuthology.orchestra.run.trial195.stdout:__sync_fetch_and_add yes 2026-02-24T16:10:07.338 INFO:teuthology.orchestra.run.trial195.stdout:__sync_synchronize yes 2026-02-24T16:10:07.358 INFO:teuthology.orchestra.run.trial195.stdout:__sync_val_compare_and_swap yes 2026-02-24T16:10:07.364 INFO:teuthology.orchestra.run.trial195.stdout:libverbs no 2026-02-24T16:10:07.374 INFO:teuthology.orchestra.run.trial195.stdout:rdmacm no 2026-02-24T16:10:07.394 INFO:teuthology.orchestra.run.trial195.stdout:asprintf() yes 2026-02-24T16:10:07.415 INFO:teuthology.orchestra.run.trial195.stdout:vasprintf() yes 2026-02-24T16:10:07.437 INFO:teuthology.orchestra.run.trial195.stdout:Linux fallocate yes 2026-02-24T16:10:07.459 INFO:teuthology.orchestra.run.trial195.stdout:POSIX fadvise yes 2026-02-24T16:10:07.480 INFO:teuthology.orchestra.run.trial195.stdout:POSIX fallocate yes 2026-02-24T16:10:07.501 INFO:teuthology.orchestra.run.trial195.stdout:sched_setaffinity(3 arg) yes 2026-02-24T16:10:07.501 INFO:teuthology.orchestra.run.trial195.stdout:sched_setaffinity(2 arg) no 2026-02-24T16:10:07.521 INFO:teuthology.orchestra.run.trial195.stdout:clock_gettime yes 2026-02-24T16:10:07.543 INFO:teuthology.orchestra.run.trial195.stdout:CLOCK_MONOTONIC yes 2026-02-24T16:10:07.564 INFO:teuthology.orchestra.run.trial195.stdout:gettimeofday yes 2026-02-24T16:10:07.586 INFO:teuthology.orchestra.run.trial195.stdout:fdatasync yes 2026-02-24T16:10:07.605 INFO:teuthology.orchestra.run.trial195.stdout:pipe() yes 2026-02-24T16:10:07.625 INFO:teuthology.orchestra.run.trial195.stdout:pipe2() yes 2026-02-24T16:10:07.647 INFO:teuthology.orchestra.run.trial195.stdout:pread() yes 2026-02-24T16:10:07.673 INFO:teuthology.orchestra.run.trial195.stdout:sync_file_range yes 2026-02-24T16:10:07.679 INFO:teuthology.orchestra.run.trial195.stdout:ASharedMemory_create no 2026-02-24T16:10:07.686 INFO:teuthology.orchestra.run.trial195.stdout:EXT4 move extent yes 2026-02-24T16:10:07.707 INFO:teuthology.orchestra.run.trial195.stdout:Linux splice(2) yes 2026-02-24T16:10:07.712 INFO:teuthology.orchestra.run.trial195.stdout:libnuma no 2026-02-24T16:10:07.733 INFO:teuthology.orchestra.run.trial195.stdout:strsep yes 2026-02-24T16:10:07.754 INFO:teuthology.orchestra.run.trial195.stdout:strcasestr yes 2026-02-24T16:10:07.763 INFO:teuthology.orchestra.run.trial195.stdout:strlcat no 2026-02-24T16:10:07.786 INFO:teuthology.orchestra.run.trial195.stdout:getopt_long_only() yes 2026-02-24T16:10:07.812 INFO:teuthology.orchestra.run.trial195.stdout:inet_aton yes 2026-02-24T16:10:07.833 INFO:teuthology.orchestra.run.trial195.stdout:socklen_t yes 2026-02-24T16:10:07.855 INFO:teuthology.orchestra.run.trial195.stdout:__thread yes 2026-02-24T16:10:07.876 INFO:teuthology.orchestra.run.trial195.stdout:RUSAGE_THREAD yes 2026-02-24T16:10:07.896 INFO:teuthology.orchestra.run.trial195.stdout:SCHED_IDLE yes 2026-02-24T16:10:07.920 INFO:teuthology.orchestra.run.trial195.stdout:TCP_NODELAY yes 2026-02-24T16:10:07.943 INFO:teuthology.orchestra.run.trial195.stdout:vsock yes 2026-02-24T16:10:07.965 INFO:teuthology.orchestra.run.trial195.stdout:Net engine window_size yes 2026-02-24T16:10:07.989 INFO:teuthology.orchestra.run.trial195.stdout:TCP_MAXSEG yes 2026-02-24T16:10:08.010 INFO:teuthology.orchestra.run.trial195.stdout:RLIMIT_MEMLOCK yes 2026-02-24T16:10:08.032 INFO:teuthology.orchestra.run.trial195.stdout:pwritev/preadv yes 2026-02-24T16:10:08.054 INFO:teuthology.orchestra.run.trial195.stdout:pwritev2/preadv2 yes 2026-02-24T16:10:08.077 INFO:teuthology.orchestra.run.trial195.stdout:IPv6 helpers yes 2026-02-24T16:10:08.089 INFO:teuthology.orchestra.run.trial195.stdout:http engine no 2026-02-24T16:10:08.141 INFO:teuthology.orchestra.run.trial195.stdout:Rados engine yes 2026-02-24T16:10:08.209 INFO:teuthology.orchestra.run.trial195.stdout:Rados Block Device engine yes 2026-02-24T16:10:08.276 INFO:teuthology.orchestra.run.trial195.stdout:rbd_poll yes 2026-02-24T16:10:08.341 INFO:teuthology.orchestra.run.trial195.stdout:rbd_invalidate_cache yes 2026-02-24T16:10:08.409 INFO:teuthology.orchestra.run.trial195.stdout:rbd_encryption_load yes 2026-02-24T16:10:08.430 INFO:teuthology.orchestra.run.trial195.stdout:setvbuf yes 2026-02-24T16:10:08.435 INFO:teuthology.orchestra.run.trial195.stdout:Gluster API engine no 2026-02-24T16:10:08.446 INFO:teuthology.orchestra.run.trial195.stdout:s390_z196_facilities no 2026-02-24T16:10:08.447 INFO:teuthology.orchestra.run.trial195.stdout:HDFS engine no 2026-02-24T16:10:08.466 INFO:teuthology.orchestra.run.trial195.stdout:MTD yes 2026-02-24T16:10:08.471 INFO:teuthology.orchestra.run.trial195.stdout:libpmem no 2026-02-24T16:10:08.471 INFO:teuthology.orchestra.run.trial195.stdout:libpmem1_5 no 2026-02-24T16:10:08.477 INFO:teuthology.orchestra.run.trial195.stdout:libpmem2 no 2026-02-24T16:10:08.477 INFO:teuthology.orchestra.run.trial195.stdout:PMDK dev-dax engine no 2026-02-24T16:10:08.477 INFO:teuthology.orchestra.run.trial195.stdout:PMDK libpmem engine no 2026-02-24T16:10:08.482 INFO:teuthology.orchestra.run.trial195.stdout:DDN's Infinite Memory Engine no 2026-02-24T16:10:08.482 INFO:teuthology.orchestra.run.trial195.stdout:iscsi engine no 2026-02-24T16:10:08.482 INFO:teuthology.orchestra.run.trial195.stdout:NBD engine no 2026-02-24T16:10:08.488 INFO:teuthology.orchestra.run.trial195.stdout:DAOS File System (dfs) Engine no 2026-02-24T16:10:08.495 INFO:teuthology.orchestra.run.trial195.stdout:NFS engine no 2026-02-24T16:10:08.495 INFO:teuthology.orchestra.run.trial195.stdout:lex/yacc for arithmetic no 2026-02-24T16:10:08.515 INFO:teuthology.orchestra.run.trial195.stdout:getmntent yes 2026-02-24T16:10:08.527 INFO:teuthology.orchestra.run.trial195.stdout:getmntinfo no 2026-02-24T16:10:08.559 INFO:teuthology.orchestra.run.trial195.stdout:Static Assert yes 2026-02-24T16:10:08.578 INFO:teuthology.orchestra.run.trial195.stdout:bool yes 2026-02-24T16:10:08.601 INFO:teuthology.orchestra.run.trial195.stdout:strndup yes 2026-02-24T16:10:08.607 INFO:teuthology.orchestra.run.trial195.stdout:Valgrind headers no 2026-02-24T16:10:08.627 INFO:teuthology.orchestra.run.trial195.stdout:Zoned block device support yes 2026-02-24T16:10:08.646 INFO:teuthology.orchestra.run.trial195.stdout:Zoned block device capacity yes 2026-02-24T16:10:08.652 INFO:teuthology.orchestra.run.trial195.stdout:libzbc engine no 2026-02-24T16:10:08.672 INFO:teuthology.orchestra.run.trial195.stdout:NVMe uring command support yes 2026-02-24T16:10:08.684 INFO:teuthology.orchestra.run.trial195.stdout:xnvme engine no 2026-02-24T16:10:08.689 INFO:teuthology.orchestra.run.trial195.stdout:isal no 2026-02-24T16:10:08.690 INFO:teuthology.orchestra.run.trial195.stdout:isal CRC64 no 2026-02-24T16:10:08.701 INFO:teuthology.orchestra.run.trial195.stdout:libblkio engine no 2026-02-24T16:10:08.701 INFO:teuthology.orchestra.run.trial195.stdout:march_armv8_a_crc_crypto no 2026-02-24T16:10:08.701 INFO:teuthology.orchestra.run.trial195.stdout:cuda no 2026-02-24T16:10:08.702 INFO:teuthology.orchestra.run.trial195.stdout:libcufile no 2026-02-24T16:10:08.723 INFO:teuthology.orchestra.run.trial195.stdout:Build march=native yes 2026-02-24T16:10:08.729 INFO:teuthology.orchestra.run.trial195.stdout:CUnit no 2026-02-24T16:10:08.749 INFO:teuthology.orchestra.run.trial195.stdout:__kernel_rwf_t yes 2026-02-24T16:10:08.768 INFO:teuthology.orchestra.run.trial195.stdout:-Wimplicit-fallthrough=2 yes 2026-02-24T16:10:08.789 INFO:teuthology.orchestra.run.trial195.stdout:-Wno-stringop-truncation yes 2026-02-24T16:10:08.809 INFO:teuthology.orchestra.run.trial195.stdout:MADV_HUGEPAGE yes 2026-02-24T16:10:08.829 INFO:teuthology.orchestra.run.trial195.stdout:gettid yes 2026-02-24T16:10:08.850 INFO:teuthology.orchestra.run.trial195.stdout:statx(2)/libc yes 2026-02-24T16:10:08.873 INFO:teuthology.orchestra.run.trial195.stdout:statx(2)/syscall yes 2026-02-24T16:10:08.875 INFO:teuthology.orchestra.run.trial195.stdout:Windows PDB generation no 2026-02-24T16:10:08.895 INFO:teuthology.orchestra.run.trial195.stdout:timerfd_create yes 2026-02-24T16:10:08.896 INFO:teuthology.orchestra.run.trial195.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2026-02-24T16:10:08.896 INFO:teuthology.orchestra.run.trial195.stdout:Lib-based ioengines dynamic no 2026-02-24T16:10:08.938 INFO:teuthology.orchestra.run.trial195.stdout:TCMalloc support yes 2026-02-24T16:10:08.940 INFO:teuthology.orchestra.run.trial195.stdout:seed_buckets 4 2026-02-24T16:10:08.984 INFO:teuthology.orchestra.run.trial195.stderr:FIO_VERSION = fio-3.41-113-g411de 2026-02-24T16:10:09.009 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crc16.o 2026-02-24T16:10:09.049 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crc32.o 2026-02-24T16:10:09.076 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crc32c-arm64.o 2026-02-24T16:10:09.120 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crc32c-intel.o 2026-02-24T16:10:09.154 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crc32c.o 2026-02-24T16:10:09.183 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crc64.o 2026-02-24T16:10:09.211 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crc7.o 2026-02-24T16:10:09.234 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/crct10dif_common.o 2026-02-24T16:10:09.262 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/fnv.o 2026-02-24T16:10:09.293 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/md5.o 2026-02-24T16:10:09.350 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/murmur3.o 2026-02-24T16:10:09.382 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/sha1.o 2026-02-24T16:10:09.480 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/sha256.o 2026-02-24T16:10:09.621 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/sha3.o 2026-02-24T16:10:09.704 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/sha512.o 2026-02-24T16:10:09.772 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/test.o 2026-02-24T16:10:09.864 INFO:teuthology.orchestra.run.trial195.stdout: CC crc/xxhash.o 2026-02-24T16:10:09.919 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/axmap.o 2026-02-24T16:10:09.990 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/bloom.o 2026-02-24T16:10:10.042 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/flist_sort.o 2026-02-24T16:10:10.088 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/gauss.o 2026-02-24T16:10:10.140 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/getrusage.o 2026-02-24T16:10:10.168 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/hweight.o 2026-02-24T16:10:10.197 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/ieee754.o 2026-02-24T16:10:10.231 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/lfsr.o 2026-02-24T16:10:10.289 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/memalign.o 2026-02-24T16:10:10.322 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/memcpy.o 2026-02-24T16:10:10.406 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/mountcheck.o 2026-02-24T16:10:10.435 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/num2str.o 2026-02-24T16:10:10.480 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/output_buffer.o 2026-02-24T16:10:10.513 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/pattern.o 2026-02-24T16:10:10.596 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/prio_tree.o 2026-02-24T16:10:10.669 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/rand.o 2026-02-24T16:10:10.743 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/rbtree.o 2026-02-24T16:10:10.814 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/strntol.o 2026-02-24T16:10:10.849 INFO:teuthology.orchestra.run.trial195.stdout: CC lib/zipf.o 2026-02-24T16:10:10.914 INFO:teuthology.orchestra.run.trial195.stdout: CC gettime.o 2026-02-24T16:10:11.071 INFO:teuthology.orchestra.run.trial195.stdout: CC ioengines.o 2026-02-24T16:10:11.207 INFO:teuthology.orchestra.run.trial195.stdout: CC init.o 2026-02-24T16:10:11.585 INFO:teuthology.orchestra.run.trial195.stdout: CC stat.o 2026-02-24T16:10:12.294 INFO:teuthology.orchestra.run.trial195.stdout: CC log.o 2026-02-24T16:10:12.366 INFO:teuthology.orchestra.run.trial195.stdout: CC time.o 2026-02-24T16:10:12.455 INFO:teuthology.orchestra.run.trial195.stdout: CC filesetup.o 2026-02-24T16:10:12.789 INFO:teuthology.orchestra.run.trial195.stdout: CC eta.o 2026-02-24T16:10:12.954 INFO:teuthology.orchestra.run.trial195.stdout: CC verify.o 2026-02-24T16:10:13.294 INFO:teuthology.orchestra.run.trial195.stdout: CC memory.o 2026-02-24T16:10:13.389 INFO:teuthology.orchestra.run.trial195.stdout: CC io_u.o 2026-02-24T16:10:13.774 INFO:teuthology.orchestra.run.trial195.stdout: CC parse.o 2026-02-24T16:10:13.995 INFO:teuthology.orchestra.run.trial195.stdout: CC fio_sem.o 2026-02-24T16:10:14.061 INFO:teuthology.orchestra.run.trial195.stdout: CC rwlock.o 2026-02-24T16:10:14.115 INFO:teuthology.orchestra.run.trial195.stdout: CC pshared.o 2026-02-24T16:10:14.158 INFO:teuthology.orchestra.run.trial195.stdout: CC options.o 2026-02-24T16:10:14.563 INFO:teuthology.orchestra.run.trial195.stdout: CC fio_shared_sem.o 2026-02-24T16:10:14.596 INFO:teuthology.orchestra.run.trial195.stdout: CC smalloc.o 2026-02-24T16:10:14.699 INFO:teuthology.orchestra.run.trial195.stdout: CC filehash.o 2026-02-24T16:10:14.786 INFO:teuthology.orchestra.run.trial195.stdout: CC profile.o 2026-02-24T16:10:14.856 INFO:teuthology.orchestra.run.trial195.stdout: CC debug.o 2026-02-24T16:10:14.885 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/cpu.o 2026-02-24T16:10:14.962 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/mmap.o 2026-02-24T16:10:15.055 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/sync.o 2026-02-24T16:10:15.175 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/null.o 2026-02-24T16:10:15.242 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/net.o 2026-02-24T16:10:15.451 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/ftruncate.o 2026-02-24T16:10:15.511 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/fileoperations.o 2026-02-24T16:10:15.603 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/exec.o 2026-02-24T16:10:15.686 INFO:teuthology.orchestra.run.trial195.stdout: CC server.o 2026-02-24T16:10:16.044 INFO:teuthology.orchestra.run.trial195.stdout: CC client.o 2026-02-24T16:10:16.321 INFO:teuthology.orchestra.run.trial195.stdout: CC iolog.o 2026-02-24T16:10:16.553 INFO:teuthology.orchestra.run.trial195.stdout: CC backend.o 2026-02-24T16:10:16.968 INFO:teuthology.orchestra.run.trial195.stdout: CC libfio.o 2026-02-24T16:10:17.074 INFO:teuthology.orchestra.run.trial195.stdout: CC flow.o 2026-02-24T16:10:17.145 INFO:teuthology.orchestra.run.trial195.stdout: CC cconv.o 2026-02-24T16:10:17.363 INFO:teuthology.orchestra.run.trial195.stdout: CC gettime-thread.o 2026-02-24T16:10:17.425 INFO:teuthology.orchestra.run.trial195.stdout: CC helpers.o 2026-02-24T16:10:17.466 INFO:teuthology.orchestra.run.trial195.stdout: CC json.o 2026-02-24T16:10:17.568 INFO:teuthology.orchestra.run.trial195.stdout: CC idletime.o 2026-02-24T16:10:17.688 INFO:teuthology.orchestra.run.trial195.stdout: CC td_error.o 2026-02-24T16:10:17.749 INFO:teuthology.orchestra.run.trial195.stdout: CC profiles/tiobench.o 2026-02-24T16:10:17.809 INFO:teuthology.orchestra.run.trial195.stdout: CC profiles/act.o 2026-02-24T16:10:17.897 INFO:teuthology.orchestra.run.trial195.stdout: CC io_u_queue.o 2026-02-24T16:10:17.933 INFO:teuthology.orchestra.run.trial195.stdout: CC filelock.o 2026-02-24T16:10:18.007 INFO:teuthology.orchestra.run.trial195.stdout: CC workqueue.o 2026-02-24T16:10:18.105 INFO:teuthology.orchestra.run.trial195.stdout: CC rate-submit.o 2026-02-24T16:10:18.193 INFO:teuthology.orchestra.run.trial195.stdout: CC optgroup.o 2026-02-24T16:10:18.226 INFO:teuthology.orchestra.run.trial195.stdout: CC helper_thread.o 2026-02-24T16:10:18.327 INFO:teuthology.orchestra.run.trial195.stdout: CC steadystate.o 2026-02-24T16:10:18.471 INFO:teuthology.orchestra.run.trial195.stdout: CC zone-dist.o 2026-02-24T16:10:18.547 INFO:teuthology.orchestra.run.trial195.stdout: CC zbd.o 2026-02-24T16:10:18.909 INFO:teuthology.orchestra.run.trial195.stdout: CC dedupe.o 2026-02-24T16:10:18.994 INFO:teuthology.orchestra.run.trial195.stdout: CC dataplacement.o 2026-02-24T16:10:19.089 INFO:teuthology.orchestra.run.trial195.stdout: CC sprandom.o 2026-02-24T16:10:19.249 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/posixaio.o 2026-02-24T16:10:19.325 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/falloc.o 2026-02-24T16:10:19.393 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/e4defrag.o 2026-02-24T16:10:19.469 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/splice.o 2026-02-24T16:10:19.568 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/asprintf.o 2026-02-24T16:10:19.595 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/strlcat.o 2026-02-24T16:10:19.628 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/mtd.o 2026-02-24T16:10:19.720 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/libmtd.o 2026-02-24T16:10:19.913 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/libmtd_legacy.o 2026-02-24T16:10:19.987 INFO:teuthology.orchestra.run.trial195.stdout: CC diskutil.o 2026-02-24T16:10:20.102 INFO:teuthology.orchestra.run.trial195.stdout: CC fifo.o 2026-02-24T16:10:20.139 INFO:teuthology.orchestra.run.trial195.stdout: CC blktrace.o 2026-02-24T16:10:20.304 INFO:teuthology.orchestra.run.trial195.stdout: CC cgroup.o 2026-02-24T16:10:20.389 INFO:teuthology.orchestra.run.trial195.stdout: CC trim.o 2026-02-24T16:10:20.459 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/sg.o 2026-02-24T16:10:20.650 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/linux-dev-lookup.o 2026-02-24T16:10:20.690 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/io_uring.o 2026-02-24T16:10:20.920 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/nvme.o 2026-02-24T16:10:21.082 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/linux-blkzoned.o 2026-02-24T16:10:21.171 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/cmdprio.o 2026-02-24T16:10:21.297 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/rados.o 2026-02-24T16:10:21.398 INFO:teuthology.orchestra.run.trial195.stdout: CC engines/rbd.o 2026-02-24T16:10:21.503 INFO:teuthology.orchestra.run.trial195.stdout: CC fio.o 2026-02-24T16:10:21.565 INFO:teuthology.orchestra.run.trial195.stdout: LINK fio 2026-02-24T16:10:21.687 INFO:teuthology.orchestra.run.trial195.stdout: CC t/log.o 2026-02-24T16:10:21.718 INFO:teuthology.orchestra.run.trial195.stdout: CC t/genzipf.o 2026-02-24T16:10:21.793 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/strcasestr.o 2026-02-24T16:10:21.813 INFO:teuthology.orchestra.run.trial195.stdout: CC oslib/strndup.o 2026-02-24T16:10:21.833 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/fio-genzipf 2026-02-24T16:10:21.892 INFO:teuthology.orchestra.run.trial195.stdout: CC t/btrace2fio.o 2026-02-24T16:10:22.094 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/fio-btrace2fio 2026-02-24T16:10:22.147 INFO:teuthology.orchestra.run.trial195.stdout: CC t/verify-state.o 2026-02-24T16:10:22.208 INFO:teuthology.orchestra.run.trial195.stdout: CC t/debug.o 2026-02-24T16:10:22.234 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/fio-verify-state 2026-02-24T16:10:22.288 INFO:teuthology.orchestra.run.trial195.stdout: CC t/stest.o 2026-02-24T16:10:22.336 INFO:teuthology.orchestra.run.trial195.stdout: CC t/arch.o 2026-02-24T16:10:22.360 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/stest 2026-02-24T16:10:22.418 INFO:teuthology.orchestra.run.trial195.stdout: CC t/ieee754.o 2026-02-24T16:10:22.447 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/ieee754 2026-02-24T16:10:22.503 INFO:teuthology.orchestra.run.trial195.stdout: CC t/axmap.o 2026-02-24T16:10:22.562 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/axmap 2026-02-24T16:10:22.615 INFO:teuthology.orchestra.run.trial195.stdout: CC t/lfsr-test.o 2026-02-24T16:10:22.672 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/lfsr-test 2026-02-24T16:10:22.728 INFO:teuthology.orchestra.run.trial195.stdout: CC t/gen-rand.o 2026-02-24T16:10:22.783 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/gen-rand 2026-02-24T16:10:22.837 INFO:teuthology.orchestra.run.trial195.stdout: CC t/memlock.o 2026-02-24T16:10:22.878 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/memlock 2026-02-24T16:10:22.932 INFO:teuthology.orchestra.run.trial195.stdout: CC t/read-to-pipe-async.o 2026-02-24T16:10:23.036 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/read-to-pipe-async 2026-02-24T16:10:23.091 INFO:teuthology.orchestra.run.trial195.stdout: CC t/io_uring.o 2026-02-24T16:10:23.365 INFO:teuthology.orchestra.run.trial195.stdout: LINK t/io_uring 2026-02-24T16:10:23.421 DEBUG:teuthology.orchestra.run.trial195:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-24T16:10:23.448 DEBUG:teuthology.orchestra.run.trial195:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:10:23.558 INFO:teuthology.orchestra.run.trial195.stderr:16:10:23 - DEBUG - cbt - Settings.cluster: 2026-02-24T16:10:23.558 INFO:teuthology.orchestra.run.trial195.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'clients': ['ubuntu@trial195.front.sepia.ceph.com'], 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'head': 'ubuntu@trial195.front.sepia.ceph.com', 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'is_teuthology': True, 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'iterations': 1, 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'mons': ['ubuntu@trial195.front.sepia.ceph.com'], 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'osds': ['ubuntu@trial195.front.sepia.ceph.com'], 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'osds_per_node': 3, 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'pid_dir': '/var/run/ceph', 2026-02-24T16:10:23.559 INFO:teuthology.orchestra.run.trial195.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2026-02-24T16:10:23.560 INFO:teuthology.orchestra.run.trial195.stderr: 'rebuild_every_test': False, 2026-02-24T16:10:23.560 INFO:teuthology.orchestra.run.trial195.stderr: 'rgws': {'ubuntu@trial195.front.sepia.ceph.com': None}, 2026-02-24T16:10:23.560 INFO:teuthology.orchestra.run.trial195.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-24T16:10:23.560 INFO:teuthology.orchestra.run.trial195.stderr: 'use_existing': True, 2026-02-24T16:10:23.560 INFO:teuthology.orchestra.run.trial195.stderr: 'user': 'ubuntu'} 2026-02-24T16:10:23.560 INFO:teuthology.orchestra.run.trial195.stderr:16:10:23 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 2026-02-24T16:10:23.560 INFO:teuthology.orchestra.run.trial195.stderr:16:10:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread; fi 2026-02-24T16:10:23.731 INFO:teuthology.orchestra.run.trial195.stderr:16:10:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2026-02-24T16:10:23.894 INFO:teuthology.orchestra.run.trial195.stderr:16:10:23 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-24T16:10:23.894 INFO:teuthology.orchestra.run.trial195.stderr:16:10:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2026-02-24T16:10:24.064 INFO:teuthology.orchestra.run.trial195.stderr:16:10:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:10:24.066 INFO:teuthology.orchestra.run.trial195.stderr:16:10:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2026-02-24T16:10:24.236 INFO:teuthology.orchestra.run.trial195.stderr:16:10:24 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:10:24.236 INFO:teuthology.orchestra.run.trial195.stderr:16:10:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:11:24.298 INFO:teuthology.orchestra.run.trial195.stderr:16:11:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:11:24.482 INFO:teuthology.orchestra.run.trial195.stderr:16:11:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:11:24.667 INFO:teuthology.orchestra.run.trial195.stderr:16:11:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2026-02-24T16:11:24.841 INFO:teuthology.orchestra.run.trial195.stderr:16:11:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial195.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 2026-02-24T16:11:25.007 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - INFO - cbt - Creating fio images... 2026-02-24T16:11:25.007 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2026-02-24T16:11:25.174 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:11:25.176 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2026-02-24T16:11:25.341 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:11:25.341 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:11:25.343 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2026-02-24T16:11:25.699 INFO:teuthology.orchestra.run.trial195.stderr:16:11:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2026-02-24T16:11:26.061 INFO:teuthology.orchestra.run.trial195.stderr:16:11:26 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2026-02-24T16:11:27.068 INFO:teuthology.orchestra.run.trial195.stderr:16:11:27 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2026-02-24T16:11:28.067 INFO:teuthology.orchestra.run.trial195.stderr:16:11:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2026-02-24T16:11:29.069 INFO:teuthology.orchestra.run.trial195.stderr:16:11:29 - INFO - cbt - Checking Health after pool creation. 2026-02-24T16:11:34.075 INFO:teuthology.orchestra.run.trial195.stderr:16:11:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:11:34.442 INFO:teuthology.orchestra.run.trial195.stderr:16:11:34 - INFO - cbt - Final Pool Health Check. 2026-02-24T16:11:39.448 INFO:teuthology.orchestra.run.trial195.stderr:16:11:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:11:39.815 INFO:teuthology.orchestra.run.trial195.stderr:16:11:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com hostname -f 2026-02-24T16:11:39.985 INFO:teuthology.orchestra.run.trial195.stderr:16:11:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial195-0 --size 4096 --pool cbt-librbdfio --order 22 2026-02-24T16:11:42.210 INFO:teuthology.orchestra.run.trial195.stderr:16:11:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial195-1 --size 4096 --pool cbt-librbdfio --order 22 2026-02-24T16:11:42.463 INFO:teuthology.orchestra.run.trial195.stderr:16:11:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:11:42.632 INFO:teuthology.orchestra.run.trial195.stderr:16:11:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:11:42.823 INFO:teuthology.orchestra.run.trial195.stderr:16:11:42 - INFO - cbt - Attempting to prefill fio images... 2026-02-24T16:11:42.823 INFO:teuthology.orchestra.run.trial195.stderr:16:11:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:11:42.824 INFO:teuthology.orchestra.run.trial195.stderr:16:11:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:12:36.158 INFO:teuthology.orchestra.run.trial195.stderr:16:12:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:12:36.160 INFO:teuthology.orchestra.run.trial195.stderr:16:12:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-24T16:13:00.504 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 2026-02-24T16:13:00.512 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433. 2026-02-24T16:13:00.512 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - INFO - cbt - Running benchmark /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2026-02-24T16:13:00.512 INFO:teuthology.orchestra.run.trial195.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 2026-02-24T16:13:00.512 INFO:teuthology.orchestra.run.trial195.stderr:{'cmd_path': '/home/ubuntu/cephtest/fio/fio', 'log_avg_msec': 100, 'norandommap': True, 'pool_profile': 'rbd', 'time': 60, 'vol_size': 4096, 'iodepth': 32, 'mode': 'randread', 'op_size': 4194304, 'osd_ra': 4096, 'procs_per_volume': 1, 'volumes_per_client': 2, 'benchmark': 'librbdfio', 'iteration': 0} == iteration 0 == 2026-02-24T16:13:00.512 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - INFO - cbt - Setting OSD Read Ahead to: 4096 2026-02-24T16:13:00.512 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-24T16:13:00.513 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2026-02-24T16:13:00.514 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2026-02-24T16:13:00.693 INFO:teuthology.orchestra.run.trial195.stderr:16:13:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sync 2026-02-24T16:13:01.036 INFO:teuthology.orchestra.run.trial195.stderr:16:13:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-24T16:13:02.219 INFO:teuthology.orchestra.run.trial195.stderr:16:13:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2026-02-24T16:13:02.407 INFO:teuthology.orchestra.run.trial195.stderr:16:13:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/ceph_settings.out 2026-02-24T16:13:27.725 INFO:teuthology.orchestra.run.trial195.stderr:16:13:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log 2026-02-24T16:13:28.090 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/collectl 2026-02-24T16:13:28.254 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:13:28.256 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf 2026-02-24T16:13:28.419 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:13:28.419 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.${pid} & done 2026-02-24T16:13:28.421 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - INFO - cbt - Running rbd fio randread test. 2026-02-24T16:13:28.421 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2026-02-24T16:13:28.421 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 2026-02-24T16:13:28.422 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2026-02-24T16:13:28.422 INFO:teuthology.orchestra.run.trial195.stderr:16:13:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 2026-02-24T16:13:32.662 INFO:teuthology.orchestra.run.trial195.stderr:16:13:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:13:32.837 INFO:teuthology.orchestra.run.trial195.stderr:16:13:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:13:33.013 INFO:teuthology.orchestra.run.trial195.stderr:16:13:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf/perf.data 2026-02-24T16:13:33.015 INFO:teuthology.orchestra.run.trial195.stderr:16:13:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.* 2026-02-24T16:13:33.017 INFO:teuthology.orchestra.run.trial195.stderr:16:13:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/historic_ops.out 2026-02-24T16:13:33.371 INFO:teuthology.orchestra.run.trial195.stderr:16:13:33 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial195.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2026-02-24T16:13:33.542 INFO:teuthology.orchestra.run.trial195.stderr:16:13:33 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial195.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 2026-02-24T16:13:33.716 INFO:teuthology.orchestra.run.trial195.stderr:16:13:33 - INFO - cbt - Convert results to json format. 2026-02-24T16:13:33.716 INFO:teuthology.orchestra.run.trial195.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2026-02-24T16:13:33.716 INFO:teuthology.orchestra.run.trial195.stdout:trial195: trial195 2026-02-24T16:13:33.717 INFO:teuthology.orchestra.run.trial195.stdout: 2026-02-24T16:13:33.717 INFO:teuthology.orchestra.run.trial195.stdout:['trial195'] 2026-02-24T16:13:33.729 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-24T16:13:33.743 DEBUG:teuthology.orchestra.run.trial195:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-24T16:13:33.768 DEBUG:teuthology.orchestra.run.trial195:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2026-02-24T16:13:33.849 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-24T16:13:33.849 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-24T16:13:33.850 DEBUG:teuthology.orchestra.run.trial195:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-24T16:13:33.906 INFO:teuthology.orchestra.run.trial195.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.0.trial195.front.sepia.ceph.com 2026-02-24T16:13:33.907 INFO:teuthology.orchestra.run.trial195.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.1.trial195.front.sepia.ceph.com 2026-02-24T16:13:33.907 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:13:33.908 DEBUG:teuthology.orchestra.run.trial195:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.0.trial195.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:13:33.967 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:13:33.967 DEBUG:teuthology.orchestra.run.trial195:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 -name 'perf_stat.*' 2026-02-24T16:13:34.024 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:13:34.024 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:13:34.025 DEBUG:teuthology.orchestra.run.trial195:> set -ex 2026-02-24T16:13:34.025 DEBUG:teuthology.orchestra.run.trial195:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.1.trial195.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:13:34.082 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:13:34.082 DEBUG:teuthology.orchestra.run.trial195:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 -name 'perf_stat.*' 2026-02-24T16:13:34.138 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:13:34.138 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:13:34.141 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-24T16:13:34.141 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-24T16:13:34.141 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-24T16:13:34.152 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-24T16:13:34.153 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-24T16:13:34.153 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial195.front.sepia.ceph.com 2026-02-24T16:13:34.153 DEBUG:teuthology.orchestra.run.trial195:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:13:34.195 DEBUG:teuthology.orchestra.run.trial195:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:13:34.251 DEBUG:teuthology.orchestra.run.trial195:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:13:34.308 INFO:teuthology.orchestra.run.trial195.stdout:"600" 2026-02-24T16:13:34.309 DEBUG:teuthology.orchestra.run.trial195:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:09:57:265981 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:13:34.367 DEBUG:teuthology.orchestra.run.trial195:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:13:34.424 DEBUG:teuthology.orchestra.run.trial195:> rm /home/ubuntu/.ssh/config 2026-02-24T16:13:34.480 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-24T16:13:34.495 DEBUG:teuthology.orchestra.run.trial195:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-24T16:13:34.596 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:13:34.589+0000 7f07e8ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:13:34.597 INFO:teuthology.orchestra.run.trial195.stderr:2026-02-24T16:13:34.594+0000 7f07e3577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:13:34.720 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-24T16:13:34.721 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-24T16:13:34.721 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-24T16:13:34.721 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:34.807 INFO:tasks.ceph.osd.0:Stopped 2026-02-24T16:13:34.808 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-24T16:13:34.808 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:34.884 INFO:tasks.ceph.osd.1:Stopped 2026-02-24T16:13:34.884 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-24T16:13:34.884 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:34.969 INFO:tasks.ceph.osd.2:Stopped 2026-02-24T16:13:34.969 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-24T16:13:34.969 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-24T16:13:34.969 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:35.003 INFO:tasks.ceph.mgr.x:Stopped 2026-02-24T16:13:35.003 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-24T16:13:35.004 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-24T16:13:35.004 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:13:35.054 INFO:tasks.ceph.mon.a:Stopped 2026-02-24T16:13:35.054 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-24T16:13:35.054 DEBUG:teuthology.orchestra.run.trial195:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-24T16:13:35.120 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:13:35.120 DEBUG:teuthology.orchestra.run.trial195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-24T16:13:35.199 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:13:35.199 DEBUG:teuthology.orchestra.run.trial195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-24T16:13:35.269 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:13:35.269 DEBUG:teuthology.orchestra.run.trial195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-24T16:13:35.342 INFO:tasks.ceph:Archiving mon data... 2026-02-24T16:13:35.343 DEBUG:teuthology.misc:Transferring archived files from trial195:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68175/data/mon.a.tgz 2026-02-24T16:13:35.344 DEBUG:teuthology.orchestra.run.trial195:> mktemp 2026-02-24T16:13:35.398 INFO:teuthology.orchestra.run.trial195.stdout:/tmp/tmp.c023tm3XgV 2026-02-24T16:13:35.399 DEBUG:teuthology.orchestra.run.trial195:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.c023tm3XgV 2026-02-24T16:13:35.488 DEBUG:teuthology.orchestra.run.trial195:> sudo chmod 0666 /tmp/tmp.c023tm3XgV 2026-02-24T16:13:35.570 DEBUG:teuthology.orchestra.remote:trial195:/tmp/tmp.c023tm3XgV is 197KB 2026-02-24T16:13:35.623 DEBUG:teuthology.orchestra.run.trial195:> rm -fr /tmp/tmp.c023tm3XgV 2026-02-24T16:13:35.639 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-24T16:13:35.639 DEBUG:teuthology.orchestra.run.trial195:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-24T16:13:35.760 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-24T16:13:35.760 INFO:tasks.ceph:Archiving crash dumps... 2026-02-24T16:13:35.762 DEBUG:teuthology.misc:Transferring archived files from trial195:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68175/remote/trial195/crash 2026-02-24T16:13:35.763 DEBUG:teuthology.orchestra.run.trial195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-24T16:13:35.823 INFO:tasks.ceph:Compressing logs... 2026-02-24T16:13:35.823 DEBUG:teuthology.orchestra.run.trial195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:13:35.888 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-24T16:13:35.888 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-24T16:13:35.888 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41701.log 2026-02-24T16:13:35.904 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2026-02-24T16:13:35.904 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47249.log 2026-02-24T16:13:35.904 INFO:teuthology.orchestra.run.trial195.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41701.log.gz 2026-02-24T16:13:35.904 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41552.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47137.log 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47249.log: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-osd.0.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47249.log.gz 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41451.log 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40989.log 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47137.log.gz 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48389.log 2026-02-24T16:13:35.905 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41451.log.gz 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40989.log: -- /var/log/ceph/ceph-client.admin.47023.log 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40989.log.gz 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr:gzip/var/log/ceph/ceph-client.admin.48389.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48389.log.gz 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47023.log.gz 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47193.log 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47469.log: /var/log/ceph/ceph-client.admin.40384.log: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41892.log 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2026-02-24T16:13:35.906 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41402.log 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47193.log: /var/log/ceph/ceph-client.admin.41892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41554.log 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47193.log.gz 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41892.log.gz 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41402.log.gz 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40289.log 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41554.log.gz 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.40568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40615.log 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gz 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.40289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40289.log.gz 2026-02-24T16:13:35.907 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.40615.log: /var/log/ceph/ceph-client.admin.41553.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40615.log.gz -5 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47818.log 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41064.log 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41064.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47818.log.gz 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41064.log.gz 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.40431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47646.log 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2026-02-24T16:13:35.908 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48388.log 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47646.log.gz 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr: 86.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.48388.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39000.log 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48388.log.gz 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph.tmp-client.admin.39000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41353.log 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39000.log.gz 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2026-02-24T16:13:35.909 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47281.log 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41353.log.gz 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47647.log 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40662.log 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47281.log.gz 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47647.log.gz 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.40662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41214.log 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40662.log.gz 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48153.log 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41239.log: /var/log/ceph/ceph-client.admin.41214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47817.log 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41214.log.gz 2026-02-24T16:13:35.910 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.48153.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48153.log.gz 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47079.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40881.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47817.log.gz 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47335.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47079.log: /var/log/ceph/ceph-client.admin.40881.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47079.log.gz 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40881.log.gz 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49045.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47335.log.gz 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47421.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.49045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2026-02-24T16:13:35.911 INFO:teuthology.orchestra.run.trial195.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.49045.log.gz 2026-02-24T16:13:35.912 INFO:teuthology.orchestra.run.trial195.stderr: 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-24T16:13:35.912 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.47421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2026-02-24T16:13:35.912 INFO:teuthology.orchestra.run.trial195.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.47421.log.gz 2026-02-24T16:13:35.912 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41843.log 2026-02-24T16:13:35.912 INFO:teuthology.orchestra.run.trial195.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2026-02-24T16:13:35.912 INFO:teuthology.orchestra.run.trial195.stderr:/var/log/ceph/ceph-client.admin.41700.log: /var/log/ceph/ceph-client.admin.41843.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2026-02-24T16:13:35.912 INFO:teuthology.orchestra.run.trial195.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41843.log.gz 2026-02-24T16:13:35.913 INFO:teuthology.orchestra.run.trial195.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-24T16:13:38.294 INFO:teuthology.orchestra.run.trial195.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-24T16:13:38.342 INFO:teuthology.orchestra.run.trial195.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-24T16:13:38.765 INFO:teuthology.orchestra.run.trial195.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-24T16:13:38.767 INFO:teuthology.orchestra.run.trial195.stderr: 2026-02-24T16:13:38.767 INFO:teuthology.orchestra.run.trial195.stderr:real 0m2.887s 2026-02-24T16:13:38.767 INFO:teuthology.orchestra.run.trial195.stderr:user 0m7.227s 2026-02-24T16:13:38.767 INFO:teuthology.orchestra.run.trial195.stderr:sys 0m0.524s 2026-02-24T16:13:38.767 INFO:tasks.ceph:Archiving logs... 2026-02-24T16:13:38.768 DEBUG:teuthology.misc:Transferring archived files from trial195:/var/log/ceph to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68175/remote/trial195/log 2026-02-24T16:13:38.769 DEBUG:teuthology.orchestra.run.trial195:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:13:39.613 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:13:39.681 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:13:39.692 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-24T16:13:39.692 DEBUG:teuthology.orchestra.run.trial195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:13:39.717 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-24T16:13:39.728 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-24T16:13:39.728 DEBUG:teuthology.orchestra.run.trial195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:13:39.771 INFO:teuthology.orchestra.run.trial195.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:13:39.796 INFO:teuthology.orchestra.run.trial195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:13:39.796 INFO:teuthology.orchestra.run.trial195.stdout:=============================================================================== 2026-02-24T16:13:39.796 INFO:teuthology.orchestra.run.trial195.stdout:^+ ntp.maxhost.io 2 7 377 45 -738us[ -738us] +/- 56ms 2026-02-24T16:13:39.796 INFO:teuthology.orchestra.run.trial195.stdout:^+ ntp.silvertree.org 1 6 377 50 +7612us[+7612us] +/- 47ms 2026-02-24T16:13:39.796 INFO:teuthology.orchestra.run.trial195.stdout:^* ip212-227-240-160.pbiaas> 2 8 377 178 -2233us[-2609us] +/- 41ms 2026-02-24T16:13:39.796 INFO:teuthology.orchestra.run.trial195.stdout:^- 23.186.168.132 2 7 355 169 -552us[ -552us] +/- 55ms 2026-02-24T16:13:39.797 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-24T16:13:39.811 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-24T16:13:39.811 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-24T16:13:39.827 DEBUG:teuthology.orchestra.run.trial195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:13:39.896 DEBUG:teuthology.orchestra.run.trial195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:13:39.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:13:39.924 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-24T16:13:39.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-24T16:13:39.957 INFO:teuthology.task.internal:Duration was 2222.757824 seconds 2026-02-24T16:13:39.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-24T16:13:39.975 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-24T16:13:39.975 DEBUG:teuthology.orchestra.run.trial195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:13:40.013 INFO:teuthology.orchestra.run.trial195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:13:40.152 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-24T16:13:40.152 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial195.front.sepia.ceph.com 2026-02-24T16:13:40.153 DEBUG:teuthology.orchestra.run.trial195:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:13:40.211 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-24T16:13:40.211 DEBUG:teuthology.orchestra.run.trial195:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:13:40.316 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-24T16:13:40.317 DEBUG:teuthology.orchestra.run.trial195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:13:40.340 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:13:40.340 INFO:teuthology.orchestra.run.trial195.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:13:40.341 INFO:teuthology.orchestra.run.trial195.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:13:40.342 INFO:teuthology.orchestra.run.trial195.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:13:40.342 INFO:teuthology.orchestra.run.trial195.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:13:40.352 INFO:teuthology.orchestra.run.trial195.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:13:40.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-24T16:13:40.370 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-24T16:13:40.371 DEBUG:teuthology.orchestra.run.trial195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:13:40.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-24T16:13:40.436 DEBUG:teuthology.orchestra.run.trial195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:13:40.483 INFO:teuthology.orchestra.run.trial195.stdout:kernel.core_pattern = core 2026-02-24T16:13:40.502 DEBUG:teuthology.orchestra.run.trial195:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:13:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:13:40.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-24T16:13:40.576 INFO:teuthology.task.internal:Transferring archived files... 2026-02-24T16:13:40.577 DEBUG:teuthology.misc:Transferring archived files from trial195:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68175/remote/trial195 2026-02-24T16:13:40.578 DEBUG:teuthology.orchestra.run.trial195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:13:40.715 INFO:teuthology.task.internal:Removing archive directory... 2026-02-24T16:13:40.716 DEBUG:teuthology.orchestra.run.trial195:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:13:40.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-24T16:13:40.762 INFO:teuthology.task.internal:Not uploading archives. 2026-02-24T16:13:40.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-24T16:13:40.797 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-24T16:13:40.798 DEBUG:teuthology.orchestra.run.trial195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:13:40.814 INFO:teuthology.orchestra.run.trial195.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:13 /home/ubuntu/cephtest 2026-02-24T16:13:40.815 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-24T16:13:40.833 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-24T16:13:40.872 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/fio_4M_rand_read} duration: 2222.757824420929 flavor: debug owner: scheduled_liucm@soko04.front.sepia.ceph.com success: true 2026-02-24T16:13:40.872 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:13:40.981 INFO:teuthology.run:pass