2026-02-23T11:37:44.912 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T11:37:44.927 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:37:46.434 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66280 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read} email: null first_in_suite: false flavor: debug job_id: '66280' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d workunit: branch: wip-shraddhaag-enable-cephadm-tests env: CRIMSON_COMPAT: '1' sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 66 sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-shraddhaag-enable-cephadm-tests suite_path: /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa suite_relpath: qa suite_repo: https://github.com/shraddhaag/ceph.git suite_sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d targets: trial127.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEthPYhMme3wvdvcXM1qyNhKwMzLH92L+xw/m8eBJwadW/cLQ137KO5x5KDIqjmVLnHikIuFc89HwpSNEa+jWXw= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-23_09:58:48 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-23T11:37:46.434 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa; will attempt to use it 2026-02-23T11:37:46.435 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa/tasks 2026-02-23T11:37:46.435 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-23T11:37:46.436 INFO:teuthology.task.internal:Checking packages... 2026-02-23T11:37:46.436 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d' 2026-02-23T11:37:46.437 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-23T11:37:46.437 INFO:teuthology.packaging:ref: None 2026-02-23T11:37:46.437 INFO:teuthology.packaging:tag: None 2026-02-23T11:37:46.437 INFO:teuthology.packaging:branch: main 2026-02-23T11:37:46.437 INFO:teuthology.packaging:sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:37:46.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-23T11:37:46.559 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5253.g5ccdd6ad 2026-02-23T11:37:46.560 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-23T11:37:46.575 INFO:teuthology.task.internal:no buildpackages task found 2026-02-23T11:37:46.575 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-23T11:37:46.588 INFO:teuthology.task.internal:Saving configuration 2026-02-23T11:37:46.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-23T11:37:46.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-23T11:37:46.693 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial127.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66280', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-23 11:34:15.819005', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEthPYhMme3wvdvcXM1qyNhKwMzLH92L+xw/m8eBJwadW/cLQ137KO5x5KDIqjmVLnHikIuFc89HwpSNEa+jWXw='} 2026-02-23T11:37:46.694 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-23T11:37:46.704 INFO:teuthology.task.internal:roles: ubuntu@trial127.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-23T11:37:46.704 INFO:teuthology.run_tasks:Running task console_log... 2026-02-23T11:37:46.891 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1844913b00>, signals=[15]) 2026-02-23T11:37:46.891 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-23T11:37:46.902 INFO:teuthology.task.internal:Opening connections... 2026-02-23T11:37:46.902 DEBUG:teuthology.task.internal:connecting to ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:37:46.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial127.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:37:46.963 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-23T11:37:46.977 DEBUG:teuthology.orchestra.run.trial127:> uname -m 2026-02-23T11:37:47.033 INFO:teuthology.orchestra.run.trial127.stdout:x86_64 2026-02-23T11:37:47.033 DEBUG:teuthology.orchestra.run.trial127:> cat /etc/os-release 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:NAME="CentOS Stream" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:VERSION="9" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:ID="centos" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:ID_LIKE="rhel fedora" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:VERSION_ID="9" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:PLATFORM_ID="platform:el9" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:ANSI_COLOR="0;31" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:LOGO="fedora-logo-icon" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:HOME_URL="https://centos.org/" 2026-02-23T11:37:47.089 INFO:teuthology.orchestra.run.trial127.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-23T11:37:47.090 INFO:teuthology.orchestra.run.trial127.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-23T11:37:47.090 INFO:teuthology.orchestra.run.trial127.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-23T11:37:47.091 INFO:teuthology.lock.ops:Updating trial127.front.sepia.ceph.com on lock server 2026-02-23T11:37:47.181 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-23T11:37:47.192 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-23T11:37:47.204 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-23T11:37:47.204 DEBUG:teuthology.orchestra.run.trial127:> test '!' -e /home/ubuntu/cephtest 2026-02-23T11:37:47.219 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-23T11:37:47.230 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-23T11:37:47.230 DEBUG:teuthology.orchestra.run.trial127:> test -z $(ls -A /var/lib/ceph) 2026-02-23T11:37:47.276 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-23T11:37:47.536 INFO:teuthology.run_tasks:Running task kernel... 2026-02-23T11:37:47.559 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-23T11:37:47.559 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-23T11:37:47.560 DEBUG:teuthology.orchestra.run.trial127:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-23T11:37:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:37:47.575 DEBUG:teuthology.orchestra.run.trial127:> uname -r 2026-02-23T11:37:47.631 INFO:teuthology.orchestra.run.trial127.stdout:5.14.0-681.el9.x86_64 2026-02-23T11:37:47.632 INFO:teuthology.task.kernel:Running kernel on trial127: 5.14.0-681.el9.x86_64 2026-02-23T11:37:47.632 DEBUG:teuthology.orchestra.run.trial127:> sudo yum install -y kernel 2026-02-23T11:37:49.395 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.9 MB 00:01 2026-02-23T11:37:51.971 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-23T11:37:56.065 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 7.8 MB 00:01 2026-02-23T11:37:57.110 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 20 kB 00:00 2026-02-23T11:37:58.242 INFO:teuthology.orchestra.run.trial127.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2026-02-23T11:38:01.448 INFO:teuthology.orchestra.run.trial127.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-23T11:38:02.458 INFO:teuthology.orchestra.run.trial127.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-23T11:38:02.458 INFO:teuthology.orchestra.run.trial127.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-23T11:38:02.477 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:38:02.478 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:38:02.478 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:38:02.501 DEBUG:teuthology.orchestra.run.trial127:> echo no | sudo yum reinstall kernel || true 2026-02-23T11:38:02.802 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 11:38:01 AM UTC. 2026-02-23T11:38:02.883 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:38:02.884 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:38:02.884 INFO:teuthology.orchestra.run.trial127.stdout: Package Architecture Version Repository Size 2026-02-23T11:38:02.884 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:38:02.884 INFO:teuthology.orchestra.run.trial127.stdout:Reinstalling: 2026-02-23T11:38:02.884 INFO:teuthology.orchestra.run.trial127.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:38:02.884 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:38:02.885 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:38:02.885 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:38:02.885 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:38:02.885 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 803 k 2026-02-23T11:38:02.885 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 0 2026-02-23T11:38:02.885 INFO:teuthology.orchestra.run.trial127.stderr:Operation aborted. 2026-02-23T11:38:02.904 INFO:teuthology.orchestra.run.trial127.stdout:Is this ok [y/N]: 2026-02-23T11:38:02.904 DEBUG:teuthology.orchestra.run.trial127:> sudo yum reinstall -y kernel || true 2026-02-23T11:38:03.201 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:38:01 AM UTC. 2026-02-23T11:38:03.281 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:38:03.281 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:38:03.281 INFO:teuthology.orchestra.run.trial127.stdout: Package Architecture Version Repository Size 2026-02-23T11:38:03.281 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:38:03.281 INFO:teuthology.orchestra.run.trial127.stdout:Reinstalling: 2026-02-23T11:38:03.281 INFO:teuthology.orchestra.run.trial127.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:38:03.281 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:38:03.282 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:38:03.282 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:38:03.282 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:38:03.282 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 803 k 2026-02-23T11:38:03.282 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 0 2026-02-23T11:38:03.282 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:38:04.457 INFO:teuthology.orchestra.run.trial127.stdout:kernel-5.14.0-681.el9.x86_64.rpm 886 kB/s | 803 kB 00:00 2026-02-23T11:38:04.458 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:38:04.458 INFO:teuthology.orchestra.run.trial127.stdout:Total 683 kB/s | 803 kB 00:01 2026-02-23T11:38:04.467 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:38:04.500 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:38:04.501 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:38:04.505 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:38:04.505 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:38:04.529 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:38:04.533 INFO:teuthology.orchestra.run.trial127.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:38:04.533 INFO:teuthology.orchestra.run.trial127.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:38:04.644 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:38:04.644 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:38:04.701 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:38:04.701 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:38:04.702 INFO:teuthology.orchestra.run.trial127.stdout:Reinstalled: 2026-02-23T11:38:04.702 INFO:teuthology.orchestra.run.trial127.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-23T11:38:04.702 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:38:04.702 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:38:04.729 DEBUG:teuthology.orchestra.run.trial127:> rpm -q kernel | sort -rV | head -n 1 2026-02-23T11:38:04.752 INFO:teuthology.orchestra.run.trial127.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-23T11:38:04.752 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-23T11:38:04.753 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-23T11:38:04.753 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-23T11:38:04.753 DEBUG:teuthology.orchestra.run.trial127:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-23T11:38:04.816 INFO:teuthology.orchestra.run.trial127.stdout:ttyS1 2026-02-23T11:38:04.841 DEBUG:teuthology.parallel:result is None 2026-02-23T11:38:04.842 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-23T11:38:04.854 INFO:teuthology.task.internal:Creating test directory... 2026-02-23T11:38:04.854 DEBUG:teuthology.orchestra.run.trial127:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-23T11:38:04.898 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-23T11:38:04.910 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-23T11:38:04.924 INFO:teuthology.task.internal:Creating archive directory... 2026-02-23T11:38:04.925 DEBUG:teuthology.orchestra.run.trial127:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-23T11:38:04.960 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-23T11:38:04.973 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-23T11:38:04.973 DEBUG:teuthology.orchestra.run.trial127:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-23T11:38:05.020 INFO:teuthology.orchestra.run.trial127.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:38:05.029 INFO:teuthology.orchestra.run.trial127.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:38:05.030 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-23T11:38:05.043 INFO:teuthology.task.internal:Configuring sudo... 2026-02-23T11:38:05.043 DEBUG:teuthology.orchestra.run.trial127:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-23T11:38:05.096 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-23T11:38:05.118 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-23T11:38:05.119 DEBUG:teuthology.orchestra.run.trial127:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-23T11:38:05.152 DEBUG:teuthology.orchestra.run.trial127:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:38:05.315 DEBUG:teuthology.orchestra.run.trial127:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:38:05.339 DEBUG:teuthology.orchestra.run.trial127:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:38:05.492 DEBUG:teuthology.orchestra.run.trial127:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:38:05.515 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:38:05.515 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-23T11:38:05.581 DEBUG:teuthology.orchestra.run.trial127:> sudo service rsyslog restart 2026-02-23T11:38:05.648 INFO:teuthology.orchestra.run.trial127.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:38:05.959 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-23T11:38:05.973 INFO:teuthology.task.internal:Starting timer... 2026-02-23T11:38:05.973 INFO:teuthology.run_tasks:Running task pcp... 2026-02-23T11:38:05.995 INFO:teuthology.run_tasks:Running task selinux... 2026-02-23T11:38:06.013 DEBUG:teuthology.orchestra.run.trial127:> sudo service auditd rotate 2026-02-23T11:38:06.041 INFO:teuthology.orchestra.run.trial127.stdout:Rotating logs: 2026-02-23T11:38:06.042 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-23T11:38:06.043 DEBUG:teuthology.orchestra.run.trial127:> /usr/sbin/getenforce 2026-02-23T11:38:06.098 INFO:teuthology.orchestra.run.trial127.stdout:Permissive 2026-02-23T11:38:06.099 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial127.front.sepia.ceph.com': 'permissive'} 2026-02-23T11:38:06.099 DEBUG:teuthology.orchestra.run.trial127:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:38:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:38:06.164 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-23T11:38:06.164 DEBUG:teuthology.orchestra.run.trial127:> sudo /usr/sbin/setenforce permissive 2026-02-23T11:38:06.231 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-23T11:38:06.253 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-23T11:38:06.267 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-23T11:38:06.268 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial127.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-23T11:40:50.707 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial127.front.sepia.ceph.com')] 2026-02-23T11:40:50.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial127.front.sepia.ceph.com' 2026-02-23T11:40:50.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial127.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:40:50.819 DEBUG:teuthology.orchestra.run.trial127:> true 2026-02-23T11:40:50.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial127.front.sepia.ceph.com' 2026-02-23T11:40:50.897 INFO:teuthology.run_tasks:Running task clock... 2026-02-23T11:40:50.915 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-23T11:40:50.915 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-23T11:40:50.916 DEBUG:teuthology.orchestra.run.trial127:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:40:50.964 INFO:teuthology.orchestra.run.trial127.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-23T11:40:50.976 INFO:teuthology.orchestra.run.trial127.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-23T11:40:50.997 INFO:teuthology.orchestra.run.trial127.stderr:sudo: ntpd: command not found 2026-02-23T11:40:51.006 INFO:teuthology.orchestra.run.trial127.stdout:506 Cannot talk to daemon 2026-02-23T11:40:51.017 INFO:teuthology.orchestra.run.trial127.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-23T11:40:51.029 INFO:teuthology.orchestra.run.trial127.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-23T11:40:51.082 INFO:teuthology.orchestra.run.trial127.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:40:51.083 INFO:teuthology.orchestra.run.trial127.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:40:51.084 INFO:teuthology.orchestra.run.trial127.stdout:=============================================================================== 2026-02-23T11:40:51.085 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:40:51.099 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:40:51.099 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:40:51.100 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:40:51.100 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:40:51.109 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:40:51.110 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-23T11:40:51.110 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:40:51.112 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:40:51.281 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:40:51.281 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:40:51.641 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:40:51.642 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:40:51.642 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:40:51.667 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-23T11:40:51.668 DEBUG:teuthology.orchestra.run.trial127:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:40:51.733 DEBUG:teuthology.orchestra.run.trial127:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:40:51.806 DEBUG:teuthology.orchestra.run.trial127:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:40:51.869 INFO:teuthology.orchestra.run.trial127.stdout:check_obsoletes = 1 2026-02-23T11:40:51.871 DEBUG:teuthology.orchestra.run.trial127:> sudo yum clean all 2026-02-23T11:40:52.031 INFO:teuthology.orchestra.run.trial127.stdout:41 files removed 2026-02-23T11:40:52.046 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-radosgw 2026-02-23T11:40:52.872 INFO:teuthology.orchestra.run.trial127.stdout:ceph packages for x86_64 134 kB/s | 92 kB 00:00 2026-02-23T11:40:53.521 INFO:teuthology.orchestra.run.trial127.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-23T11:40:54.055 INFO:teuthology.orchestra.run.trial127.stdout:ceph source packages 4.0 kB/s | 2.1 kB 00:00 2026-02-23T11:40:54.980 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.9 MB 00:00 2026-02-23T11:40:58.034 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-23T11:41:01.468 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 7.8 MB 00:01 2026-02-23T11:41:03.329 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-23T11:41:03.906 INFO:teuthology.orchestra.run.trial127.stdout:Extra Packages for Enterprise Linux 39 MB/s | 20 MB 00:00 2026-02-23T11:41:07.155 INFO:teuthology.orchestra.run.trial127.stdout:lab-extras 1.6 MB/s | 50 kB 00:00 2026-02-23T11:41:08.348 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:41:08.349 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:41:08.350 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:41:08.350 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:41:08.350 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:41:08.350 INFO:teuthology.orchestra.run.trial127.stdout: ceph-radosgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 11 M 2026-02-23T11:41:08.350 INFO:teuthology.orchestra.run.trial127.stdout:Upgrading: 2026-02-23T11:41:08.350 INFO:teuthology.orchestra.run.trial127.stdout: librados2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 3.3 M 2026-02-23T11:41:08.350 INFO:teuthology.orchestra.run.trial127.stdout: librbd1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.7 M 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: ceph-base x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.1 M 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 21 M 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: ceph-selinux x86_64 2:20.3.0-5439.ge371433f.el9 ceph 25 k 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 860 k 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-23T11:41:08.351 INFO:teuthology.orchestra.run.trial127.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: libradosstriper1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 263 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: librgw2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.6 M 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-argparse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 45 k 2026-02-23T11:41:08.352 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 186 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: python3-cephfs x86_64 2:20.3.0-5439.ge371433f.el9 ceph 204 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: python3-rados x86_64 2:20.3.0-5439.ge371433f.el9 ceph 324 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: python3-rbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 305 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: python3-rgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 100 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout:Installing weak dependencies: 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-23T11:41:08.353 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:08.354 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:41:08.354 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:41:08.354 INFO:teuthology.orchestra.run.trial127.stdout:Install 32 Packages 2026-02-23T11:41:08.354 INFO:teuthology.orchestra.run.trial127.stdout:Upgrade 2 Packages 2026-02-23T11:41:08.354 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:08.355 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 60 M 2026-02-23T11:41:08.355 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:41:11.027 INFO:teuthology.orchestra.run.trial127.stdout:(1/34): ceph-base-20.3.0-5439.ge371433f.el9.x86 2.1 MB/s | 5.1 MB 00:02 2026-02-23T11:41:11.093 INFO:teuthology.orchestra.run.trial127.stdout:(2/34): ceph-selinux-20.3.0-5439.ge371433f.el9. 382 kB/s | 25 kB 00:00 2026-02-23T11:41:11.509 INFO:teuthology.orchestra.run.trial127.stdout:(3/34): libcephfs2-20.3.0-5439.ge371433f.el9.x8 2.0 MB/s | 860 kB 00:00 2026-02-23T11:41:11.651 INFO:teuthology.orchestra.run.trial127.stdout:(4/34): libradosstriper1-20.3.0-5439.ge371433f. 1.8 MB/s | 263 kB 00:00 2026-02-23T11:41:14.809 INFO:teuthology.orchestra.run.trial127.stdout:(5/34): librgw2-20.3.0-5439.ge371433f.el9.x86_6 1.8 MB/s | 5.6 MB 00:03 2026-02-23T11:41:14.875 INFO:teuthology.orchestra.run.trial127.stdout:(6/34): python3-ceph-argparse-20.3.0-5439.ge371 685 kB/s | 45 kB 00:00 2026-02-23T11:41:15.008 INFO:teuthology.orchestra.run.trial127.stdout:(7/34): python3-ceph-common-20.3.0-5439.ge37143 1.4 MB/s | 186 kB 00:00 2026-02-23T11:41:15.142 INFO:teuthology.orchestra.run.trial127.stdout:(8/34): python3-cephfs-20.3.0-5439.ge371433f.el 1.5 MB/s | 204 kB 00:00 2026-02-23T11:41:15.414 INFO:teuthology.orchestra.run.trial127.stdout:(9/34): python3-rados-20.3.0-5439.ge371433f.el9 1.2 MB/s | 324 kB 00:00 2026-02-23T11:41:15.431 INFO:teuthology.orchestra.run.trial127.stdout:(10/34): ceph-radosgw-20.3.0-5439.ge371433f.el9 1.6 MB/s | 11 MB 00:06 2026-02-23T11:41:15.571 INFO:teuthology.orchestra.run.trial127.stdout:(11/34): python3-rgw-20.3.0-5439.ge371433f.el9. 718 kB/s | 100 kB 00:00 2026-02-23T11:41:15.623 INFO:teuthology.orchestra.run.trial127.stdout:(12/34): python3-rbd-20.3.0-5439.ge371433f.el9. 1.4 MB/s | 305 kB 00:00 2026-02-23T11:41:15.633 INFO:teuthology.orchestra.run.trial127.stdout:(13/34): mailcap-2.1.49-5.el9.noarch.rpm 528 kB/s | 33 kB 00:00 2026-02-23T11:41:15.686 INFO:teuthology.orchestra.run.trial127.stdout:(14/34): pciutils-3.7.0-7.el9.x86_64.rpm 1.4 MB/s | 93 kB 00:00 2026-02-23T11:41:16.043 INFO:teuthology.orchestra.run.trial127.stdout:(15/34): boost-program-options-1.75.0-13.el9.x8 254 kB/s | 104 kB 00:00 2026-02-23T11:41:16.139 INFO:teuthology.orchestra.run.trial127.stdout:(16/34): libnbd-1.20.3-4.el9.x86_64.rpm 362 kB/s | 164 kB 00:00 2026-02-23T11:41:16.220 INFO:teuthology.orchestra.run.trial127.stdout:(17/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 906 kB/s | 160 kB 00:00 2026-02-23T11:41:16.232 INFO:teuthology.orchestra.run.trial127.stdout:(18/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 486 kB/s | 45 kB 00:00 2026-02-23T11:41:16.410 INFO:teuthology.orchestra.run.trial127.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2026-02-23T11:41:16.460 INFO:teuthology.orchestra.run.trial127.stdout:(20/34): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2026-02-23T11:41:16.516 INFO:teuthology.orchestra.run.trial127.stdout:(21/34): qatlib-25.08.0-2.el9.x86_64.rpm 2.2 MB/s | 240 kB 00:00 2026-02-23T11:41:16.539 INFO:teuthology.orchestra.run.trial127.stdout:(22/34): qatlib-service-25.08.0-2.el9.x86_64.rp 464 kB/s | 37 kB 00:00 2026-02-23T11:41:16.601 INFO:teuthology.orchestra.run.trial127.stdout:(23/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 778 kB/s | 66 kB 00:00 2026-02-23T11:41:16.695 INFO:teuthology.orchestra.run.trial127.stdout:(24/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2026-02-23T11:41:16.721 INFO:teuthology.orchestra.run.trial127.stdout:(25/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 935 kB/s | 25 kB 00:00 2026-02-23T11:41:16.748 INFO:teuthology.orchestra.run.trial127.stdout:(26/34): liboath-2.6.12-1.el9.x86_64.rpm 1.8 MB/s | 49 kB 00:00 2026-02-23T11:41:16.775 INFO:teuthology.orchestra.run.trial127.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 2.4 MB/s | 67 kB 00:00 2026-02-23T11:41:16.827 INFO:teuthology.orchestra.run.trial127.stdout:(28/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2026-02-23T11:41:16.832 INFO:teuthology.orchestra.run.trial127.stdout:(29/34): libarrow-9.0.0-15.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2026-02-23T11:41:16.858 INFO:teuthology.orchestra.run.trial127.stdout:(30/34): re2-20211101-20.el9.x86_64.rpm 6.2 MB/s | 191 kB 00:00 2026-02-23T11:41:16.864 INFO:teuthology.orchestra.run.trial127.stdout:(31/34): thrift-0.15.0-4.el9.x86_64.rpm 51 MB/s | 1.6 MB 00:00 2026-02-23T11:41:18.522 INFO:teuthology.orchestra.run.trial127.stdout:(32/34): ceph-common-20.3.0-5439.ge371433f.el9. 2.1 MB/s | 21 MB 00:09 2026-02-23T11:41:19.671 INFO:teuthology.orchestra.run.trial127.stdout:(33/34): librbd1-20.3.0-5439.ge371433f.el9.x86_ 982 kB/s | 2.7 MB 00:02 2026-02-23T11:41:20.031 INFO:teuthology.orchestra.run.trial127.stdout:(34/34): librados2-20.3.0-5439.ge371433f.el9.x8 1.0 MB/s | 3.3 MB 00:03 2026-02-23T11:41:20.032 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:41:20.033 INFO:teuthology.orchestra.run.trial127.stdout:Total 5.2 MB/s | 60 MB 00:11 2026-02-23T11:41:20.120 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:41:20.138 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:41:20.138 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:41:20.384 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:41:20.384 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:41:20.665 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:41:20.807 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-23T11:41:20.811 INFO:teuthology.orchestra.run.trial127.stdout: Upgrading : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:41:20.842 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:41:20.879 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T11:41:20.880 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:41:20.892 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:41:20.900 INFO:teuthology.orchestra.run.trial127.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-23T11:41:20.902 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-23T11:41:20.905 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-23T11:41:20.910 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-23T11:41:20.921 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 9/36 2026-02-23T11:41:20.934 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T11:41:20.934 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T11:41:20.958 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T11:41:20.962 INFO:teuthology.orchestra.run.trial127.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-23T11:41:20.999 INFO:teuthology.orchestra.run.trial127.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-23T11:41:21.003 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-23T11:41:21.009 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-23T11:41:21.012 INFO:teuthology.orchestra.run.trial127.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-23T11:41:21.017 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-23T11:41:21.133 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-23T11:41:21.134 INFO:teuthology.orchestra.run.trial127.stdout: Upgrading : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T11:41:21.160 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T11:41:21.162 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 20/36 2026-02-23T11:41:21.168 INFO:teuthology.orchestra.run.trial127.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-23T11:41:21.417 INFO:teuthology.orchestra.run.trial127.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-23T11:41:21.418 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T11:41:21.434 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T11:41:21.440 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 24/36 2026-02-23T11:41:21.445 INFO:teuthology.orchestra.run.trial127.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-23T11:41:21.448 INFO:teuthology.orchestra.run.trial127.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-23T11:41:21.448 INFO:teuthology.orchestra.run.trial127.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T11:41:21.466 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T11:41:21.470 INFO:teuthology.orchestra.run.trial127.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-23T11:41:21.505 INFO:teuthology.orchestra.run.trial127.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-23T11:41:21.507 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 30/36 2026-02-23T11:41:22.451 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:41:22.453 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:41:22.714 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:41:22.717 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T11:41:22.758 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T11:41:22.758 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-23T11:41:22.759 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-23T11:41:22.759 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:22.761 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:41:27.294 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /sys 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /proc 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /mnt 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /var/tmp 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /home 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /root 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /tmp 2026-02-23T11:41:27.295 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:27.788 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T11:41:27.811 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T11:41:27.811 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:41:27.812 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-23T11:41:27.812 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:41:27.812 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:41:27.812 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:27.813 INFO:teuthology.orchestra.run.trial127.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T11:41:27.825 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T11:41:27.825 INFO:teuthology.orchestra.run.trial127.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:41:27.876 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:41:27.876 INFO:teuthology.orchestra.run.trial127.stdout:Creating group 'qat' with GID 992. 2026-02-23T11:41:27.877 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:28.182 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 1/36 2026-02-23T11:41:28.182 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:41:28.182 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T11:41:28.182 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:41:28.182 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 5/36 2026-02-23T11:41:28.182 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 6/36 2026-02-23T11:41:28.182 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 7/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 8/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 9/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 11/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 12/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 13/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-23T11:41:28.183 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-23T11:41:28.184 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-23T11:41:28.185 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-23T11:41:28.185 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-23T11:41:28.185 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:41:28.185 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-23T11:41:28.185 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 35/36 2026-02-23T11:41:28.242 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:41:28.242 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:28.242 INFO:teuthology.orchestra.run.trial127.stdout:Upgraded: 2026-02-23T11:41:28.242 INFO:teuthology.orchestra.run.trial127.stdout: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.243 INFO:teuthology.orchestra.run.trial127.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-23T11:41:28.244 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: re2-1:20211101-20.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:28.245 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:41:28.284 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-test 2026-02-23T11:41:28.603 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:21 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:41:28.686 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:41:28.687 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:41:28.687 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:41:28.687 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout: ceph-test x86_64 2:20.3.0-5439.ge371433f.el9 ceph 126 M 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout: libcephsqlite x86_64 2:20.3.0-5439.ge371433f.el9 ceph 153 k 2026-02-23T11:41:28.688 INFO:teuthology.orchestra.run.trial127.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout:Install 12 Packages 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:41:28.689 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 131 M 2026-02-23T11:41:28.690 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 977 M 2026-02-23T11:41:28.690 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:41:29.418 INFO:teuthology.orchestra.run.trial127.stdout:(1/12): libcephsqlite-20.3.0-5439.ge371433f.el9 316 kB/s | 153 kB 00:00 2026-02-23T11:41:29.836 INFO:teuthology.orchestra.run.trial127.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 55 kB/s | 23 kB 00:00 2026-02-23T11:41:30.092 INFO:teuthology.orchestra.run.trial127.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 368 kB/s | 94 kB 00:00 2026-02-23T11:41:30.950 INFO:teuthology.orchestra.run.trial127.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 612 kB/s | 525 kB 00:00 2026-02-23T11:41:31.701 INFO:teuthology.orchestra.run.trial127.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 235 kB/s | 176 kB 00:00 2026-02-23T11:41:31.779 INFO:teuthology.orchestra.run.trial127.stdout:(6/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 755 kB/s | 2.1 MB 00:02 2026-02-23T11:41:32.461 INFO:teuthology.orchestra.run.trial127.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 306 kB/s | 233 kB 00:00 2026-02-23T11:41:32.707 INFO:teuthology.orchestra.run.trial127.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2026-02-23T11:41:33.258 INFO:teuthology.orchestra.run.trial127.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 381 kB/s | 303 kB 00:00 2026-02-23T11:41:33.294 INFO:teuthology.orchestra.run.trial127.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 3.4 MB/s | 122 kB 00:00 2026-02-23T11:41:33.330 INFO:teuthology.orchestra.run.trial127.stdout:(11/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 102 kB/s | 64 kB 00:00 2026-02-23T11:43:14.931 INFO:teuthology.orchestra.run.trial127.stdout:(12/12): ceph-test-20.3.0-5439.ge371433f.el9.x8 1.2 MB/s | 126 MB 01:45 2026-02-23T11:43:14.934 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:43:14.934 INFO:teuthology.orchestra.run.trial127.stdout:Total 1.2 MB/s | 131 MB 01:46 2026-02-23T11:43:15.006 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:43:15.017 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:43:15.017 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:43:15.442 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:43:15.443 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:43:15.896 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:43:15.905 INFO:teuthology.orchestra.run.trial127.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-23T11:43:15.913 INFO:teuthology.orchestra.run.trial127.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-23T11:43:15.918 INFO:teuthology.orchestra.run.trial127.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-23T11:43:15.922 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-23T11:43:15.925 INFO:teuthology.orchestra.run.trial127.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-23T11:43:15.929 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-23T11:43:15.930 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T11:43:15.947 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T11:43:15.949 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-23T11:43:15.961 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-23T11:43:15.969 INFO:teuthology.orchestra.run.trial127.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-23T11:43:15.970 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T11:43:23.102 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T11:43:23.121 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T11:43:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T11:43:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 1/12 2026-02-23T11:43:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2/12 2026-02-23T11:43:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-23T11:43:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-23T11:43:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-23T11:43:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-23T11:43:23.928 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-23T11:43:23.928 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-23T11:43:23.928 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-23T11:43:23.928 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-23T11:43:23.928 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-23T11:43:23.981 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-23T11:43:23.982 INFO:teuthology.orchestra.run.trial127.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-23T11:43:23.983 INFO:teuthology.orchestra.run.trial127.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-23T11:43:23.983 INFO:teuthology.orchestra.run.trial127.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-23T11:43:23.983 INFO:teuthology.orchestra.run.trial127.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-23T11:43:23.983 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:43:23.983 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:43:24.023 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph 2026-02-23T11:43:24.331 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:02:17 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:43:24.415 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout: ceph x86_64 2:20.3.0-5439.ge371433f.el9 ceph 6.5 k 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mds x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.0 M 2026-02-23T11:43:24.417 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr x86_64 2:20.3.0-5439.ge371433f.el9 ceph 937 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 298 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mon x86_64 2:20.3.0-5439.ge371433f.el9 ceph 4.9 M 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 164 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-classic x86_64 2:20.3.0-5439.ge371433f.el9 ceph 15 M 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson x86_64 2:20.3.0-5439.ge371433f.el9 ceph 87 M 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-23T11:43:24.418 INFO:teuthology.orchestra.run.trial127.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-collections 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-23T11:43:24.419 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-23T11:43:24.420 INFO:teuthology.orchestra.run.trial127.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout:Install 42 Packages 2026-02-23T11:43:24.421 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:43:24.422 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 114 M 2026-02-23T11:43:24.422 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 961 M 2026-02-23T11:43:24.422 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:43:25.216 INFO:teuthology.orchestra.run.trial127.stdout:(1/42): ceph-20.3.0-5439.ge371433f.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-23T11:43:26.859 INFO:teuthology.orchestra.run.trial127.stdout:(2/42): ceph-mgr-20.3.0-5439.ge371433f.el9.x86_ 485 kB/s | 937 kB 00:01 2026-02-23T11:43:27.200 INFO:teuthology.orchestra.run.trial127.stdout:(3/42): ceph-osd-20.3.0-5439.ge371433f.el9.x86_ 481 kB/s | 164 kB 00:00 2026-02-23T11:43:28.027 INFO:teuthology.orchestra.run.trial127.stdout:(4/42): ceph-mds-20.3.0-5439.ge371433f.el9.x86_ 668 kB/s | 2.0 MB 00:03 2026-02-23T11:43:31.162 INFO:teuthology.orchestra.run.trial127.stdout:(5/42): ceph-mon-20.3.0-5439.ge371433f.el9.x86_ 838 kB/s | 4.9 MB 00:05 2026-02-23T11:43:31.428 INFO:teuthology.orchestra.run.trial127.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5439.ge371 1.1 MB/s | 298 kB 00:00 2026-02-23T11:43:32.023 INFO:teuthology.orchestra.run.trial127.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 68 kB/s | 40 kB 00:00 2026-02-23T11:43:32.655 INFO:teuthology.orchestra.run.trial127.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 114 kB/s | 72 kB 00:00 2026-02-23T11:43:33.423 INFO:teuthology.orchestra.run.trial127.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 329 kB/s | 253 kB 00:00 2026-02-23T11:43:34.374 INFO:teuthology.orchestra.run.trial127.stdout:(10/42): python3-cryptography-36.0.1-5.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2026-02-23T11:43:35.046 INFO:teuthology.orchestra.run.trial127.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 159 kB/s | 106 kB 00:00 2026-02-23T11:43:35.753 INFO:teuthology.orchestra.run.trial127.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 191 kB/s | 135 kB 00:00 2026-02-23T11:43:36.465 INFO:teuthology.orchestra.run.trial127.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 212 kB/s | 150 kB 00:00 2026-02-23T11:43:37.161 INFO:teuthology.orchestra.run.trial127.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 182 kB/s | 126 kB 00:00 2026-02-23T11:43:37.704 INFO:teuthology.orchestra.run.trial127.stdout:(15/42): ceph-osd-classic-20.3.0-5439.ge371433f 1.4 MB/s | 15 MB 00:10 2026-02-23T11:43:37.918 INFO:teuthology.orchestra.run.trial127.stdout:(16/42): python3-urllib3-1.26.5-7.el9.noarch.rp 288 kB/s | 218 kB 00:00 2026-02-23T11:43:38.437 INFO:teuthology.orchestra.run.trial127.stdout:(17/42): unzip-6.0-59.el9.x86_64.rpm 248 kB/s | 182 kB 00:00 2026-02-23T11:43:38.704 INFO:teuthology.orchestra.run.trial127.stdout:(18/42): zip-3.0-35.el9.x86_64.rpm 338 kB/s | 266 kB 00:00 2026-02-23T11:43:39.168 INFO:teuthology.orchestra.run.trial127.stdout:(19/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 338 kB/s | 246 kB 00:00 2026-02-23T11:43:39.337 INFO:teuthology.orchestra.run.trial127.stdout:(20/42): lua-5.4.4-4.el9.x86_64.rpm 298 kB/s | 188 kB 00:00 2026-02-23T11:43:39.410 INFO:teuthology.orchestra.run.trial127.stdout:(21/42): python3-libstoragemgmt-1.10.1-1.el9.x8 728 kB/s | 177 kB 00:00 2026-02-23T11:43:39.495 INFO:teuthology.orchestra.run.trial127.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 494 kB/s | 42 kB 00:00 2026-02-23T11:43:39.499 INFO:teuthology.orchestra.run.trial127.stdout:(23/42): python3-packaging-20.9-5.el9.noarch.rp 476 kB/s | 77 kB 00:00 2026-02-23T11:43:39.528 INFO:teuthology.orchestra.run.trial127.stdout:(24/42): luarocks-3.9.2-5.el9.noarch.rpm 5.1 MB/s | 151 kB 00:00 2026-02-23T11:43:39.532 INFO:teuthology.orchestra.run.trial127.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 7.9 MB/s | 29 kB 00:00 2026-02-23T11:43:39.536 INFO:teuthology.orchestra.run.trial127.stdout:(26/42): python3-backports-tarfile-1.2.0-1.el9. 15 MB/s | 60 kB 00:00 2026-02-23T11:43:39.540 INFO:teuthology.orchestra.run.trial127.stdout:(27/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 12 MB/s | 43 kB 00:00 2026-02-23T11:43:39.547 INFO:teuthology.orchestra.run.trial127.stdout:(28/42): python3-cheroot-10.0.1-4.el9.noarch.rp 25 MB/s | 173 kB 00:00 2026-02-23T11:43:39.554 INFO:teuthology.orchestra.run.trial127.stdout:(29/42): python3-cherrypy-18.6.1-2.el9.noarch.r 48 MB/s | 358 kB 00:00 2026-02-23T11:43:39.558 INFO:teuthology.orchestra.run.trial127.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 2.9 MB/s | 11 kB 00:00 2026-02-23T11:43:39.562 INFO:teuthology.orchestra.run.trial127.stdout:(31/42): python3-jaraco-classes-3.2.1-5.el9.noa 4.8 MB/s | 18 kB 00:00 2026-02-23T11:43:39.565 INFO:teuthology.orchestra.run.trial127.stdout:(32/42): python3-jaraco-collections-3.0.0-8.el9 6.2 MB/s | 23 kB 00:00 2026-02-23T11:43:39.569 INFO:teuthology.orchestra.run.trial127.stdout:(33/42): python3-jaraco-context-6.0.1-3.el9.noa 5.2 MB/s | 20 kB 00:00 2026-02-23T11:43:39.573 INFO:teuthology.orchestra.run.trial127.stdout:(34/42): python3-jaraco-functools-3.5.0-2.el9.n 5.3 MB/s | 19 kB 00:00 2026-02-23T11:43:39.577 INFO:teuthology.orchestra.run.trial127.stdout:(35/42): python3-jaraco-text-4.0.0-2.el9.noarch 7.1 MB/s | 26 kB 00:00 2026-02-23T11:43:39.581 INFO:teuthology.orchestra.run.trial127.stdout:(36/42): python3-more-itertools-8.12.0-2.el9.no 19 MB/s | 79 kB 00:00 2026-02-23T11:43:39.585 INFO:teuthology.orchestra.run.trial127.stdout:(37/42): python3-portend-3.1.0-2.el9.noarch.rpm 4.3 MB/s | 16 kB 00:00 2026-02-23T11:43:39.589 INFO:teuthology.orchestra.run.trial127.stdout:(38/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 22 MB/s | 90 kB 00:00 2026-02-23T11:43:39.593 INFO:teuthology.orchestra.run.trial127.stdout:(39/42): python3-tempora-5.0.0-2.el9.noarch.rpm 9.1 MB/s | 36 kB 00:00 2026-02-23T11:43:39.597 INFO:teuthology.orchestra.run.trial127.stdout:(40/42): python3-zc-lockfile-2.0-10.el9.noarch. 5.3 MB/s | 20 kB 00:00 2026-02-23T11:43:39.737 INFO:teuthology.orchestra.run.trial127.stdout:(41/42): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2026-02-23T11:44:29.858 INFO:teuthology.orchestra.run.trial127.stdout:(42/42): ceph-osd-crimson-20.3.0-5439.ge371433f 1.4 MB/s | 87 MB 01:01 2026-02-23T11:44:29.862 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:44:29.862 INFO:teuthology.orchestra.run.trial127.stdout:Total 1.7 MB/s | 114 MB 01:05 2026-02-23T11:44:29.985 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:44:30.005 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:44:30.005 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:44:30.399 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:44:30.399 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:44:30.836 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:44:30.838 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-23T11:44:30.843 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-23T11:44:30.848 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-23T11:44:30.850 INFO:teuthology.orchestra.run.trial127.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-23T11:44:30.856 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-23T11:44:30.879 INFO:teuthology.orchestra.run.trial127.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-23T11:44:30.883 INFO:teuthology.orchestra.run.trial127.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-23T11:44:30.888 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-23T11:44:30.892 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-23T11:44:30.899 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-23T11:44:30.902 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-23T11:44:30.908 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-23T11:44:30.914 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-23T11:44:30.917 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-23T11:44:30.923 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-23T11:44:30.926 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-23T11:44:30.933 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-23T11:44:30.937 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-23T11:44:30.959 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-23T11:44:30.972 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-23T11:44:30.977 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-23T11:44:30.986 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-23T11:44:30.993 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T11:44:31.006 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-23T11:44:31.022 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-23T11:44:31.097 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-23T11:44:31.105 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-23T11:44:31.128 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-23T11:44:31.157 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-23T11:44:31.163 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-23T11:44:31.216 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-23T11:44:31.257 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 32/42 2026-02-23T11:44:31.258 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T11:44:31.280 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T11:44:31.281 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:44:31.281 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-23T11:44:31.281 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:44:31.281 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:44:31.281 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:31.287 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-23T11:44:31.302 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:44:31.303 INFO:teuthology.orchestra.run.trial127.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-23T11:44:31.303 INFO:teuthology.orchestra.run.trial127.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-23T11:44:31.303 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:31.311 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:44:31.333 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:44:31.333 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-23T11:44:31.333 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:31.937 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-23T11:44:37.303 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 37/42 2026-02-23T11:44:37.313 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 38/42 2026-02-23T11:44:37.314 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T11:44:37.338 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T11:44:37.338 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:44:37.338 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-23T11:44:37.338 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:44:37.338 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:44:37.338 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:37.544 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T11:44:37.564 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T11:44:37.564 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:44:37.564 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-23T11:44:37.564 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:44:37.564 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:44:37.564 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:37.648 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T11:44:37.667 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T11:44:37.667 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:44:37.667 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-23T11:44:37.668 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:44:37.668 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:44:37.668 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:37.668 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T11:44:37.674 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T11:44:37.696 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 1/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 3/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 4/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 5/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 6/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 7/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 8/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-23T11:44:38.602 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-23T11:44:38.603 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-23T11:44:38.604 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-23T11:44:38.604 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-23T11:44:38.604 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-23T11:44:38.661 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-23T11:44:38.661 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:38.661 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:44:38.661 INFO:teuthology.orchestra.run.trial127.stdout: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: lua-5.4.4-4.el9.x86_64 2026-02-23T11:44:38.662 INFO:teuthology.orchestra.run.trial127.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-23T11:44:38.663 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-ply-3.11-14.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-23T11:44:38.664 INFO:teuthology.orchestra.run.trial127.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-23T11:44:38.665 INFO:teuthology.orchestra.run.trial127.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-23T11:44:38.665 INFO:teuthology.orchestra.run.trial127.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-23T11:44:38.665 INFO:teuthology.orchestra.run.trial127.stdout: unzip-6.0-59.el9.x86_64 2026-02-23T11:44:38.665 INFO:teuthology.orchestra.run.trial127.stdout: zip-3.0-35.el9.x86_64 2026-02-23T11:44:38.665 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:38.665 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:44:38.706 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-base 2026-02-23T11:44:39.017 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:03:32 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:44:39.081 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:44:39.098 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:44:39.099 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:44:39.099 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:44:39.120 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install cephadm 2026-02-23T11:44:39.425 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:03:32 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:44:39.507 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:44:39.507 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:39.507 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:44:39.507 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:39.507 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:44:39.507 INFO:teuthology.orchestra.run.trial127.stdout: cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 1.0 M 2026-02-23T11:44:39.507 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:39.508 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:44:39.508 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:39.508 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:44:39.508 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:39.508 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 1.0 M 2026-02-23T11:44:39.508 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 1.0 M 2026-02-23T11:44:39.508 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:44:40.540 INFO:teuthology.orchestra.run.trial127.stdout:cephadm-20.3.0-5439.ge371433f.el9.noarch.rpm 1.0 MB/s | 1.0 MB 00:01 2026-02-23T11:44:40.540 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:44:40.540 INFO:teuthology.orchestra.run.trial127.stdout:Total 1.0 MB/s | 1.0 MB 00:01 2026-02-23T11:44:40.541 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:44:40.543 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:44:40.543 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:44:40.547 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:44:40.547 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:44:40.566 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:44:40.625 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:44:40.638 INFO:teuthology.orchestra.run.trial127.stdout: Installing : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:44:40.704 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:44:40.760 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:44:40.760 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:40.761 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:44:40.761 INFO:teuthology.orchestra.run.trial127.stdout: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:44:40.761 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:40.761 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:44:40.786 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T11:44:41.095 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:03:34 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:44:41.177 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:44:41.177 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:41.177 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5439.ge371433f.el9 ceph 149 k 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 149 k 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 455 k 2026-02-23T11:44:41.178 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:44:41.547 INFO:teuthology.orchestra.run.trial127.stdout:ceph-immutable-object-cache-20.3.0-5439.ge37143 403 kB/s | 149 kB 00:00 2026-02-23T11:44:41.547 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:44:41.547 INFO:teuthology.orchestra.run.trial127.stdout:Total 402 kB/s | 149 kB 00:00 2026-02-23T11:44:41.547 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:44:41.552 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:44:41.552 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:44:41.578 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:44:41.578 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:44:41.628 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:44:41.629 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:44:41.652 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:44:41.652 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:44:41.652 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-23T11:44:41.652 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:41.967 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:44:41.967 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:41.968 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:44:41.968 INFO:teuthology.orchestra.run.trial127.stdout: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:44:41.968 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:41.968 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:44:41.995 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr 2026-02-23T11:44:42.303 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:03:35 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:44:42.367 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:44:42.385 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:44:42.386 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:44:42.386 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:44:42.407 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T11:44:42.714 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:03:35 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:44:42.796 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 51 M 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 57 k 2026-02-23T11:44:42.798 INFO:teuthology.orchestra.run.trial127.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 17 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:44:42.799 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:44:42.800 INFO:teuthology.orchestra.run.trial127.stdout:Install 13 Packages 2026-02-23T11:44:42.800 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:44:42.800 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 55 M 2026-02-23T11:44:42.800 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 262 M 2026-02-23T11:44:42.800 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:44:43.597 INFO:teuthology.orchestra.run.trial127.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5439.ge37 53 kB/s | 17 kB 00:00 2026-02-23T11:44:43.649 INFO:teuthology.orchestra.run.trial127.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5439.ge3 151 kB/s | 57 kB 00:00 2026-02-23T11:44:44.234 INFO:teuthology.orchestra.run.trial127.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 75 kB/s | 48 kB 00:00 2026-02-23T11:44:44.361 INFO:teuthology.orchestra.run.trial127.stdout:(4/13): protobuf-compiler-3.14.0-17.el9.x86_64. 6.6 MB/s | 862 kB 00:00 2026-02-23T11:44:44.521 INFO:teuthology.orchestra.run.trial127.stdout:(5/13): python3-protobuf-3.14.0-17.el9.noarch.r 307 kB/s | 267 kB 00:00 2026-02-23T11:44:44.623 INFO:teuthology.orchestra.run.trial127.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 188 kB/s | 19 kB 00:00 2026-02-23T11:44:44.744 INFO:teuthology.orchestra.run.trial127.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2026-02-23T11:44:44.765 INFO:teuthology.orchestra.run.trial127.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 6.9 MB/s | 144 kB 00:00 2026-02-23T11:44:44.785 INFO:teuthology.orchestra.run.trial127.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 1.5 MB/s | 31 kB 00:00 2026-02-23T11:44:44.803 INFO:teuthology.orchestra.run.trial127.stdout:(10/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.2 MB/s | 551 kB 00:00 2026-02-23T11:44:44.805 INFO:teuthology.orchestra.run.trial127.stdout:(11/13): python3-routes-2.5.1-5.el9.noarch.rpm 8.9 MB/s | 188 kB 00:00 2026-02-23T11:44:44.824 INFO:teuthology.orchestra.run.trial127.stdout:(12/13): python3-xmltodict-0.12.0-15.el9.noarch 1.1 MB/s | 22 kB 00:00 2026-02-23T11:45:02.235 INFO:teuthology.orchestra.run.trial127.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5439.ge37143 2.7 MB/s | 51 MB 00:18 2026-02-23T11:45:02.238 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:02.238 INFO:teuthology.orchestra.run.trial127.stdout:Total 2.8 MB/s | 55 MB 00:19 2026-02-23T11:45:02.266 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:02.275 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:02.276 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:02.488 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:02.488 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:02.716 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:02.721 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-23T11:45:02.732 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-23T11:45:02.734 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-23T11:45:02.762 INFO:teuthology.orchestra.run.trial127.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-23T11:45:02.805 INFO:teuthology.orchestra.run.trial127.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-23T11:45:02.813 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-23T11:45:02.841 INFO:teuthology.orchestra.run.trial127.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-23T11:45:02.851 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-23T11:45:02.858 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-23T11:45:02.860 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-23T11:45:02.870 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 11/13 2026-02-23T11:45:05.743 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 12/13 2026-02-23T11:45:05.750 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T11:45:06.044 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T11:45:06.044 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 1/13 2026-02-23T11:45:06.044 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 2/13 2026-02-23T11:45:06.044 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 3/13 2026-02-23T11:45:06.044 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-23T11:45:06.045 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-23T11:45:06.100 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-23T11:45:06.100 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:06.100 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:06.100 INFO:teuthology.orchestra.run.trial127.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-23T11:45:06.101 INFO:teuthology.orchestra.run.trial127.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-23T11:45:06.102 INFO:teuthology.orchestra.run.trial127.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-23T11:45:06.102 INFO:teuthology.orchestra.run.trial127.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-23T11:45:06.102 INFO:teuthology.orchestra.run.trial127.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-23T11:45:06.102 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:06.102 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:06.137 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T11:45:06.451 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:03:59 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:06.534 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:06.535 INFO:teuthology.orchestra.run.trial127.stdout:======================================================================================= 2026-02-23T11:45:06.535 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:45:06.535 INFO:teuthology.orchestra.run.trial127.stdout:======================================================================================= 2026-02-23T11:45:06.535 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:06.535 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 7.4 M 2026-02-23T11:45:06.535 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:45:06.535 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-23T11:45:06.536 INFO:teuthology.orchestra.run.trial127.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout:======================================================================================= 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout:Install 12 Packages 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 43 M 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 234 M 2026-02-23T11:45:06.537 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:06.808 INFO:teuthology.orchestra.run.trial127.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 1.4 MB/s | 184 kB 00:00 2026-02-23T11:45:06.875 INFO:teuthology.orchestra.run.trial127.stdout:(2/12): libgfortran-11.5.0-14.el9.x86_64.rpm 4.0 MB/s | 794 kB 00:00 2026-02-23T11:45:07.374 INFO:teuthology.orchestra.run.trial127.stdout:(3/12): flexiblas-3.0.4-9.el9.x86_64.rpm 52 kB/s | 30 kB 00:00 2026-02-23T11:45:07.853 INFO:teuthology.orchestra.run.trial127.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 31 kB/s | 15 kB 00:00 2026-02-23T11:45:07.921 INFO:teuthology.orchestra.run.trial127.stdout:(5/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2026-02-23T11:45:08.432 INFO:teuthology.orchestra.run.trial127.stdout:(6/12): openblas-0.3.29-1.el9.x86_64.rpm 73 kB/s | 42 kB 00:00 2026-02-23T11:45:09.033 INFO:teuthology.orchestra.run.trial127.stdout:(7/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 4.8 MB/s | 5.3 MB 00:01 2026-02-23T11:45:09.194 INFO:teuthology.orchestra.run.trial127.stdout:(8/12): python3-devel-3.9.25-3.el9.x86_64.rpm 321 kB/s | 244 kB 00:00 2026-02-23T11:45:10.022 INFO:teuthology.orchestra.run.trial127.stdout:(9/12): python3-numpy-f2py-1.23.5-2.el9.x86_64. 534 kB/s | 442 kB 00:00 2026-02-23T11:45:10.158 INFO:teuthology.orchestra.run.trial127.stdout:(10/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 5.5 MB/s | 6.1 MB 00:01 2026-02-23T11:45:11.436 INFO:teuthology.orchestra.run.trial127.stdout:(11/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 14 MB/s | 19 MB 00:01 2026-02-23T11:45:14.403 INFO:teuthology.orchestra.run.trial127.stdout:(12/12): ceph-mgr-diskprediction-local-20.3.0-5 982 kB/s | 7.4 MB 00:07 2026-02-23T11:45:14.403 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:14.404 INFO:teuthology.orchestra.run.trial127.stdout:Total 5.4 MB/s | 43 MB 00:07 2026-02-23T11:45:14.563 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:14.576 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:14.576 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:14.778 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:14.778 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:14.995 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:14.998 INFO:teuthology.orchestra.run.trial127.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-23T11:45:15.004 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-23T11:45:15.038 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-23T11:45:15.042 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-23T11:45:15.102 INFO:teuthology.orchestra.run.trial127.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-23T11:45:15.104 INFO:teuthology.orchestra.run.trial127.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-23T11:45:15.125 INFO:teuthology.orchestra.run.trial127.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-23T11:45:15.444 INFO:teuthology.orchestra.run.trial127.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-23T11:45:15.516 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-23T11:45:16.177 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T11:45:16.631 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-23T11:45:16.632 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T11:45:16.911 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T11:45:16.911 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 1/12 2026-02-23T11:45:16.911 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-23T11:45:16.911 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-23T11:45:16.911 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-23T11:45:16.912 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-23T11:45:16.912 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-23T11:45:16.912 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-23T11:45:16.912 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-23T11:45:16.912 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-23T11:45:16.912 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T11:45:16.912 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-23T11:45:16.974 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-23T11:45:16.975 INFO:teuthology.orchestra.run.trial127.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-23T11:45:16.976 INFO:teuthology.orchestra.run.trial127.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-23T11:45:16.976 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-23T11:45:16.976 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-23T11:45:16.976 INFO:teuthology.orchestra.run.trial127.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-23T11:45:16.976 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:16.976 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:17.014 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-rook 2026-02-23T11:45:17.324 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:10 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:17.408 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:17.409 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:17.409 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:45:17.409 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-rook noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 50 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-23T11:45:17.410 INFO:teuthology.orchestra.run.trial127.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout:Install 10 Packages 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 2.0 M 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 25 M 2026-02-23T11:45:17.411 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:17.672 INFO:teuthology.orchestra.run.trial127.stdout:(1/10): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2026-02-23T11:45:17.695 INFO:teuthology.orchestra.run.trial127.stdout:(2/10): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 279 kB 00:00 2026-02-23T11:45:17.698 INFO:teuthology.orchestra.run.trial127.stdout:(3/10): python3-requests-oauthlib-1.3.0-12.el9. 2.0 MB/s | 54 kB 00:00 2026-02-23T11:45:17.832 INFO:teuthology.orchestra.run.trial127.stdout:(4/10): python3-certifi-2023.05.07-4.el9.noarch 105 kB/s | 14 kB 00:00 2026-02-23T11:45:17.846 INFO:teuthology.orchestra.run.trial127.stdout:(5/10): python3-cachetools-4.2.4-1.el9.noarch.r 213 kB/s | 32 kB 00:00 2026-02-23T11:45:17.964 INFO:teuthology.orchestra.run.trial127.stdout:(6/10): python3-google-auth-2.45.0-1.el9.noarch 1.9 MB/s | 254 kB 00:00 2026-02-23T11:45:17.983 INFO:teuthology.orchestra.run.trial127.stdout:(7/10): ceph-mgr-rook-20.3.0-5439.ge371433f.el9 121 kB/s | 50 kB 00:00 2026-02-23T11:45:17.998 INFO:teuthology.orchestra.run.trial127.stdout:(8/10): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2026-02-23T11:45:18.026 INFO:teuthology.orchestra.run.trial127.stdout:(9/10): python3-kubernetes-26.1.0-3.el9.noarch. 5.7 MB/s | 1.0 MB 00:00 2026-02-23T11:45:18.174 INFO:teuthology.orchestra.run.trial127.stdout:(10/10): python3-websocket-client-1.2.3-2.el9.n 469 kB/s | 90 kB 00:00 2026-02-23T11:45:18.176 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:18.177 INFO:teuthology.orchestra.run.trial127.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2026-02-23T11:45:18.205 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:18.213 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:18.213 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:18.253 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:18.253 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:18.332 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:18.382 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/10 2026-02-23T11:45:18.395 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/10 2026-02-23T11:45:18.409 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-23T11:45:18.414 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-23T11:45:18.421 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-23T11:45:18.461 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-23T11:45:18.473 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T11:45:18.790 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-23T11:45:18.805 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-23T11:45:18.806 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 1/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 2/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 3/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-23T11:45:18.916 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-23T11:45:18.974 INFO:teuthology.orchestra.run.trial127.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-23T11:45:18.975 INFO:teuthology.orchestra.run.trial127.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-23T11:45:18.975 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:18.975 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:19.006 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T11:45:19.314 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:12 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:19.396 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:19.397 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:19.397 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 184 k 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-23T11:45:19.398 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:19.399 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:19.399 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:19.399 INFO:teuthology.orchestra.run.trial127.stdout:Install 7 Packages 2026-02-23T11:45:19.399 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:19.399 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 7.1 M 2026-02-23T11:45:19.399 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 34 M 2026-02-23T11:45:19.399 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:20.045 INFO:teuthology.orchestra.run.trial127.stdout:(1/7): ceph-mgr-cephadm-20.3.0-5439.ge371433f.e 384 kB/s | 184 kB 00:00 2026-02-23T11:45:20.250 INFO:teuthology.orchestra.run.trial127.stdout:(2/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 363 kB/s | 249 kB 00:00 2026-02-23T11:45:20.465 INFO:teuthology.orchestra.run.trial127.stdout:(3/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 83 kB/s | 35 kB 00:00 2026-02-23T11:45:20.508 INFO:teuthology.orchestra.run.trial127.stdout:(4/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2026-02-23T11:45:20.543 INFO:teuthology.orchestra.run.trial127.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 2.4 MB/s | 86 kB 00:00 2026-02-23T11:45:20.614 INFO:teuthology.orchestra.run.trial127.stdout:(6/7): python3-natsort-7.1.1-5.el9.noarch.rpm 386 kB/s | 58 kB 00:00 2026-02-23T11:45:20.868 INFO:teuthology.orchestra.run.trial127.stdout:(7/7): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2026-02-23T11:45:20.870 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:20.870 INFO:teuthology.orchestra.run.trial127.stdout:Total 4.8 MB/s | 7.1 MB 00:01 2026-02-23T11:45:20.909 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:20.915 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:20.916 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:20.989 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:20.989 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:21.079 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:21.128 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-23T11:45:21.137 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-23T11:45:21.143 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-23T11:45:21.362 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T11:45:21.385 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-23T11:45:21.409 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-23T11:45:21.410 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T11:45:21.520 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T11:45:21.520 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/7 2026-02-23T11:45:21.520 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-23T11:45:21.520 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-23T11:45:21.520 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T11:45:21.520 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-23T11:45:21.520 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:21.577 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:21.608 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd 2026-02-23T11:45:21.920 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:14 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:21.984 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:22.003 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:22.003 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:22.003 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:22.025 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-classic 2026-02-23T11:45:22.331 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:15 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:22.396 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:22.414 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:22.415 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:22.415 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:22.436 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-fuse 2026-02-23T11:45:22.743 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:15 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:22.827 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout: ceph-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 914 k 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:22.828 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:22.829 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:45:22.829 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:22.829 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 914 k 2026-02-23T11:45:22.829 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 2.9 M 2026-02-23T11:45:22.829 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:23.606 INFO:teuthology.orchestra.run.trial127.stdout:ceph-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.1 MB/s | 914 kB 00:00 2026-02-23T11:45:23.607 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:23.607 INFO:teuthology.orchestra.run.trial127.stdout:Total 1.1 MB/s | 914 kB 00:00 2026-02-23T11:45:23.607 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:23.612 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:23.612 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:23.645 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:23.645 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:23.734 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:23.751 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:23.972 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:24.028 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:24.028 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:24.028 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:24.028 INFO:teuthology.orchestra.run.trial127.stdout: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:45:24.028 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:24.028 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:24.056 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-volume 2026-02-23T11:45:24.365 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:17 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:24.448 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout: ceph-volume noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 301 k 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:24.449 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:24.450 INFO:teuthology.orchestra.run.trial127.stdout:Install 2 Packages 2026-02-23T11:45:24.450 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:24.450 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 652 k 2026-02-23T11:45:24.450 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 2.2 M 2026-02-23T11:45:24.450 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:25.011 INFO:teuthology.orchestra.run.trial127.stdout:(1/2): ceph-volume-20.3.0-5439.ge371433f.el9.no 632 kB/s | 301 kB 00:00 2026-02-23T11:45:25.338 INFO:teuthology.orchestra.run.trial127.stdout:(2/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 437 kB/s | 351 kB 00:00 2026-02-23T11:45:25.340 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:25.341 INFO:teuthology.orchestra.run.trial127.stdout:Total 731 kB/s | 652 kB 00:00 2026-02-23T11:45:25.351 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:25.358 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:25.358 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:25.402 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:25.402 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:25.482 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:25.545 INFO:teuthology.orchestra.run.trial127.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-23T11:45:25.547 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T11:45:25.563 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T11:45:25.563 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:45:25.564 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-23T11:45:25.564 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:25.793 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 1/2 2026-02-23T11:45:25.851 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-23T11:45:25.852 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:25.852 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:25.852 INFO:teuthology.orchestra.run.trial127.stdout: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-23T11:45:25.852 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:25.852 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:25.882 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados-devel 2026-02-23T11:45:26.195 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:19 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:26.278 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:26.278 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout: librados-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 124 k 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:26.279 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 124 k 2026-02-23T11:45:26.280 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 434 k 2026-02-23T11:45:26.280 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:26.756 INFO:teuthology.orchestra.run.trial127.stdout:librados-devel-20.3.0-5439.ge371433f.el9.x86_64 259 kB/s | 124 kB 00:00 2026-02-23T11:45:26.757 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:26.757 INFO:teuthology.orchestra.run.trial127.stdout:Total 259 kB/s | 124 kB 00:00 2026-02-23T11:45:26.757 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:26.759 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:26.759 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:26.790 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:26.791 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:26.844 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:26.861 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:26.941 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:26.999 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:26.999 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:26.999 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:26.999 INFO:teuthology.orchestra.run.trial127.stdout: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:45:26.999 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:26.999 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:27.027 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs2 2026-02-23T11:45:27.339 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:20 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:27.403 INFO:teuthology.orchestra.run.trial127.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:27.421 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:27.422 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:27.422 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:27.444 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs-devel 2026-02-23T11:45:27.752 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:20 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:27.836 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:27.836 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:27.836 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-23T11:45:27.836 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 37 k 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 26 k 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout:Install 2 Packages 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 64 k 2026-02-23T11:45:27.837 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 231 k 2026-02-23T11:45:27.838 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:28.166 INFO:teuthology.orchestra.run.trial127.stdout:(1/2): libcephfs-proxy2-20.3.0-5439.ge371433f.e 80 kB/s | 26 kB 00:00 2026-02-23T11:45:28.169 INFO:teuthology.orchestra.run.trial127.stdout:(2/2): libcephfs-devel-20.3.0-5439.ge371433f.el 112 kB/s | 37 kB 00:00 2026-02-23T11:45:28.169 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:28.169 INFO:teuthology.orchestra.run.trial127.stdout:Total 191 kB/s | 64 kB 00:00 2026-02-23T11:45:28.169 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:28.173 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:28.173 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:28.204 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:28.204 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:28.252 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:28.253 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:45:28.268 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:45:28.284 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:45:28.336 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:45:28.337 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:45:28.394 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:45:28.394 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:28.394 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:28.394 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:45:28.394 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:45:28.395 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:28.395 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:28.423 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados2 2026-02-23T11:45:28.729 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:21 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:28.793 INFO:teuthology.orchestra.run.trial127.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:28.811 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:28.812 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:28.812 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:28.833 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librbd1 2026-02-23T11:45:29.141 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:22 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:29.206 INFO:teuthology.orchestra.run.trial127.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:29.224 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:29.225 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:29.225 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:29.246 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rados 2026-02-23T11:45:29.557 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:22 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:29.622 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:29.641 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:29.642 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:29.642 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:29.664 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rgw 2026-02-23T11:45:29.972 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:22 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:30.035 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:30.054 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:30.054 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:30.054 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:30.075 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-cephfs 2026-02-23T11:45:30.385 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:23 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:30.450 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:30.468 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:30.469 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:30.469 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:30.490 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rbd 2026-02-23T11:45:30.804 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:23 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:30.868 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:30.886 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:30.887 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:30.887 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:30.908 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-fuse 2026-02-23T11:45:31.222 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:24 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:31.306 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:31.306 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:31.306 INFO:teuthology.orchestra.run.trial127.stdout: Package Architecture Version Repository Size 2026-02-23T11:45:31.306 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:31.306 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:31.306 INFO:teuthology.orchestra.run.trial127.stdout: rbd-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 86 k 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 86 k 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 236 k 2026-02-23T11:45:31.307 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:31.687 INFO:teuthology.orchestra.run.trial127.stdout:rbd-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 227 kB/s | 86 kB 00:00 2026-02-23T11:45:31.688 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:31.688 INFO:teuthology.orchestra.run.trial127.stdout:Total 226 kB/s | 86 kB 00:00 2026-02-23T11:45:31.688 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:31.692 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:31.692 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:31.724 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:31.724 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:31.774 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:31.791 INFO:teuthology.orchestra.run.trial127.stdout: Installing : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:31.874 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:31.935 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:31.935 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:31.935 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:31.935 INFO:teuthology.orchestra.run.trial127.stdout: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:45:31.935 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:31.936 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:31.962 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-mirror 2026-02-23T11:45:32.269 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:25 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:32.351 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:32.351 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:32.351 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-23T11:45:32.351 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:32.351 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout: rbd-mirror x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.6 M 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 2.6 M 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 11 M 2026-02-23T11:45:32.352 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:33.887 INFO:teuthology.orchestra.run.trial127.stdout:rbd-mirror-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.7 MB/s | 2.6 MB 00:01 2026-02-23T11:45:33.887 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:33.887 INFO:teuthology.orchestra.run.trial127.stdout:Total 1.7 MB/s | 2.6 MB 00:01 2026-02-23T11:45:33.887 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:33.892 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:33.892 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:33.930 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:33.930 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:34.092 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:34.093 INFO:teuthology.orchestra.run.trial127.stdout: Installing : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:34.116 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:34.116 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:45:34.116 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-23T11:45:34.116 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:45:34.116 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:45:34.116 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:34.414 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:34.415 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:34.415 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:34.415 INFO:teuthology.orchestra.run.trial127.stdout: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:45:34.415 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:34.415 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:34.443 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-nbd 2026-02-23T11:45:34.759 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:27 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:34.843 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:34.843 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:34.843 INFO:teuthology.orchestra.run.trial127.stdout: Package Architecture Version Repository Size 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout: rbd-nbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 166 k 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 166 k 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 496 k 2026-02-23T11:45:34.844 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:45:35.329 INFO:teuthology.orchestra.run.trial127.stdout:rbd-nbd-20.3.0-5439.ge371433f.el9.x86_64.rpm 344 kB/s | 166 kB 00:00 2026-02-23T11:45:35.329 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:45:35.329 INFO:teuthology.orchestra.run.trial127.stdout:Total 343 kB/s | 166 kB 00:00 2026-02-23T11:45:35.329 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:45:35.334 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:45:35.334 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:45:35.365 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:45:35.366 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:45:35.422 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:45:35.439 INFO:teuthology.orchestra.run.trial127.stdout: Installing : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:35.524 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:35.583 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:45:35.584 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:35.584 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:45:35.584 INFO:teuthology.orchestra.run.trial127.stdout: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:45:35.584 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:35.584 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:35.614 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-crimson 2026-02-23T11:45:35.931 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:28 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:35.997 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:45:36.015 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:36.016 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:45:36.016 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:45:36.039 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-23T11:45:36.355 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:04:29 ago on Mon 23 Feb 2026 11:41:07 AM UTC. 2026-02-23T11:45:36.438 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5439.ge371433f.el9 ceph 429 M 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 429 M 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 2.6 G 2026-02-23T11:45:36.439 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:49:56.411 INFO:teuthology.orchestra.run.trial127.stdout:ceph-osd-crimson-debuginfo-20.3.0-5439.ge371433 1.7 MB/s | 429 MB 04:19 2026-02-23T11:49:56.418 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:49:56.418 INFO:teuthology.orchestra.run.trial127.stdout:Total 1.7 MB/s | 429 MB 04:19 2026-02-23T11:49:56.418 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:49:56.418 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:49:56.418 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:49:57.693 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:49:57.694 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:50:21.092 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:50:21.109 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:50:22.983 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:50:23.042 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:50:23.042 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:23.042 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:50:23.042 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:23.042 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:23.042 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:23.115 DEBUG:teuthology.parallel:result is None 2026-02-23T11:50:23.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:50:23.301 DEBUG:teuthology.orchestra.run.trial127:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:50:23.322 INFO:teuthology.orchestra.run.trial127.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:50:23.323 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:50:23.323 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:50:23.326 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-23T11:50:23.326 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:23.326 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-23T11:50:23.389 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-23T11:50:23.389 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:23.389 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/usr/bin/daemon-helper 2026-02-23T11:50:23.455 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-23T11:50:23.518 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-23T11:50:23.519 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:23.519 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-23T11:50:23.584 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-23T11:50:23.648 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-23T11:50:23.648 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:23.648 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/usr/bin/stdin-killer 2026-02-23T11:50:23.712 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-23T11:50:23.776 INFO:teuthology.run_tasks:Running task pexec... 2026-02-23T11:50:23.799 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-23T11:50:23.799 DEBUG:teuthology.orchestra.run.trial127:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-23T11:50:23.819 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:50:23.819 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-23T11:50:23.870 DEBUG:teuthology.parallel:result is None 2026-02-23T11:50:23.870 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:50:23.892 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:50:23.892 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:50:23.892 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:50:23.892 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:50:23.903 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:50:23.903 INFO:teuthology.task.install:extra packages: [] 2026-02-23T11:50:23.904 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:50:23.904 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:50:24.066 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:50:24.066 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:50:24.418 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:50:24.418 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:24.418 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:50:24.444 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-23T11:50:24.444 DEBUG:teuthology.orchestra.run.trial127:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:50:24.511 DEBUG:teuthology.orchestra.run.trial127:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:50:24.576 DEBUG:teuthology.orchestra.run.trial127:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:50:24.632 INFO:teuthology.orchestra.run.trial127.stdout:check_obsoletes = 1 2026-02-23T11:50:24.641 DEBUG:teuthology.orchestra.run.trial127:> sudo yum clean all 2026-02-23T11:50:24.799 INFO:teuthology.orchestra.run.trial127.stdout:56 files removed 2026-02-23T11:50:24.813 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-radosgw 2026-02-23T11:50:25.600 INFO:teuthology.orchestra.run.trial127.stdout:ceph packages for x86_64 143 kB/s | 92 kB 00:00 2026-02-23T11:50:26.211 INFO:teuthology.orchestra.run.trial127.stdout:ceph noarch packages 41 kB/s | 24 kB 00:00 2026-02-23T11:50:26.641 INFO:teuthology.orchestra.run.trial127.stdout:ceph source packages 5.1 kB/s | 2.1 kB 00:00 2026-02-23T11:50:27.563 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.9 MB 00:00 2026-02-23T11:50:30.749 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-23T11:50:34.191 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 7.8 MB 00:01 2026-02-23T11:50:35.099 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - Extras packages 109 kB/s | 20 kB 00:00 2026-02-23T11:50:36.007 INFO:teuthology.orchestra.run.trial127.stdout:Extra Packages for Enterprise Linux 24 MB/s | 20 MB 00:00 2026-02-23T11:50:39.214 INFO:teuthology.orchestra.run.trial127.stdout:lab-extras 2.9 MB/s | 50 kB 00:00 2026-02-23T11:50:40.395 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:40.413 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:40.414 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:40.414 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:40.439 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-test 2026-02-23T11:50:40.748 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:40.813 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:40.832 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:40.832 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:40.832 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:40.854 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph 2026-02-23T11:50:41.161 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:41.226 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:41.245 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:41.245 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:41.246 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:41.268 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-base 2026-02-23T11:50:41.584 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:41.649 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:41.667 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:41.668 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:41.668 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:41.690 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install cephadm 2026-02-23T11:50:42.002 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:42.069 INFO:teuthology.orchestra.run.trial127.stdout:Package cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:50:42.087 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:42.088 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:42.088 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:42.110 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T11:50:42.419 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:42.486 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:42.505 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:42.505 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:42.505 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:42.528 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr 2026-02-23T11:50:42.843 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:42.907 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:42.925 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:42.926 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:42.926 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:42.949 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T11:50:43.269 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:43.334 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:50:43.352 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:43.353 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:43.353 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:43.375 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T11:50:43.685 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:43.750 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:50:43.768 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:43.768 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:43.769 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:43.790 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-rook 2026-02-23T11:50:44.103 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:44.169 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:50:44.187 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:44.188 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:44.188 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:44.209 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T11:50:44.519 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:44.584 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:50:44.602 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:44.602 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:44.602 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:44.624 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd 2026-02-23T11:50:44.939 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:45.006 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:45.024 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:45.025 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:45.025 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:45.047 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-classic 2026-02-23T11:50:45.354 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:45.420 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:45.439 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:45.439 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:45.439 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:45.460 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-fuse 2026-02-23T11:50:45.772 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:45.837 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:45.856 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:45.856 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:45.857 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:45.879 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-volume 2026-02-23T11:50:46.188 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:46.253 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:50:46.272 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:46.272 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:46.273 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:46.295 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados-devel 2026-02-23T11:50:46.602 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:46.668 INFO:teuthology.orchestra.run.trial127.stdout:Package librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:46.686 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:46.687 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:46.687 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:46.709 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs2 2026-02-23T11:50:47.016 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:47.080 INFO:teuthology.orchestra.run.trial127.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:47.098 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:47.099 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:47.099 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:47.121 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs-devel 2026-02-23T11:50:47.428 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:47.493 INFO:teuthology.orchestra.run.trial127.stdout:Package libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:47.512 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:47.512 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:47.512 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:47.534 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados2 2026-02-23T11:50:47.844 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:47.909 INFO:teuthology.orchestra.run.trial127.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:47.928 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:47.928 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:47.929 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:47.950 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librbd1 2026-02-23T11:50:48.262 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:48.328 INFO:teuthology.orchestra.run.trial127.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:48.346 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:48.347 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:48.347 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:48.369 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rados 2026-02-23T11:50:48.678 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:48.743 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:48.761 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:48.762 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:48.762 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:48.784 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rgw 2026-02-23T11:50:49.094 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:49.158 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:49.177 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:49.177 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:49.177 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:49.200 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-cephfs 2026-02-23T11:50:49.510 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:49.575 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:49.593 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:49.594 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:49.594 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:49.615 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rbd 2026-02-23T11:50:49.921 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:49.985 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:50.004 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:50.004 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:50.004 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:50.025 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-fuse 2026-02-23T11:50:50.334 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:50.403 INFO:teuthology.orchestra.run.trial127.stdout:Package rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:50.421 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:50.422 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:50.422 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:50.447 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-mirror 2026-02-23T11:50:50.757 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:50.821 INFO:teuthology.orchestra.run.trial127.stdout:Package rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:50.839 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:50.840 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:50.840 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:50.861 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-nbd 2026-02-23T11:50:51.173 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:50:51.238 INFO:teuthology.orchestra.run.trial127.stdout:Package rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:51.256 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:50:51.257 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-23T11:50:51.257 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:50:51.282 DEBUG:teuthology.parallel:result is None 2026-02-23T11:50:51.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:50:51.454 DEBUG:teuthology.orchestra.run.trial127:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:50:51.475 INFO:teuthology.orchestra.run.trial127.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:50:51.476 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:50:51.476 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:50:51.479 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-23T11:50:51.479 INFO:teuthology.run_tasks:Running task ceph... 2026-02-23T11:50:51.580 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-23T11:50:51.580 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 777 /var/log/ceph 2026-02-23T11:50:51.606 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-23T11:50:51.607 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-23T11:50:51.669 INFO:tasks.ceph:Creating extra log directories... 2026-02-23T11:50:51.669 DEBUG:teuthology.orchestra.run.trial127:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-23T11:50:51.745 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-23T11:50:51.745 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-23T11:50:51.746 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66280', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '66280', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'workunit': {'branch': 'wip-shraddhaag-enable-cephadm-tests', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 49, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 66, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-shraddhaag-enable-cephadm-tests', 'suite_path': '/home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/shraddhaag/ceph.git', 'suite_sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'targets': {'trial127.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEthPYhMme3wvdvcXM1qyNhKwMzLH92L+xw/m8eBJwadW/cLQ137KO5x5KDIqjmVLnHikIuFc89HwpSNEa+jWXw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-23_09:58:48', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-23T11:50:51.747 DEBUG:teuthology.orchestra.run.trial127:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-23T11:50:51.790 DEBUG:teuthology.orchestra.run.trial127:> sudo install -d -m0777 -- /var/run/ceph 2026-02-23T11:50:51.854 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:51.855 DEBUG:teuthology.orchestra.run.trial127:> dd if=/scratch_devs of=/dev/stdout 2026-02-23T11:50:51.912 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-23T11:50:51.912 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_1 2026-02-23T11:50:51.971 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-23T11:50:51.971 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:50:51.971 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 907 Links: 1 2026-02-23T11:50:51.971 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:50:51.971 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:50:51.971 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-23 11:45:34.257514969 +0000 2026-02-23T11:50:51.971 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-23 11:40:41.115178272 +0000 2026-02-23T11:50:51.972 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-23 11:40:41.115178272 +0000 2026-02-23T11:50:51.972 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-23 11:40:41.115178272 +0000 2026-02-23T11:50:51.972 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-23T11:50:52.033 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-23T11:50:52.033 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-23T11:50:52.034 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.00016069 s, 3.2 MB/s 2026-02-23T11:50:52.035 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-23T11:50:52.092 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_2 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 938 Links: 1 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-23 11:45:34.258514954 +0000 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-23 11:40:41.343174289 +0000 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-23 11:40:41.343174289 +0000 2026-02-23T11:50:52.148 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-23 11:40:41.343174289 +0000 2026-02-23T11:50:52.148 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-23T11:50:52.210 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-23T11:50:52.210 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-23T11:50:52.210 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.000147765 s, 3.5 MB/s 2026-02-23T11:50:52.211 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-23T11:50:52.267 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_3 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 964 Links: 1 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-23 11:45:34.258514954 +0000 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-23 11:40:41.583170096 +0000 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-23 11:40:41.583170096 +0000 2026-02-23T11:50:52.323 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-23 11:40:41.583170096 +0000 2026-02-23T11:50:52.323 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-23T11:50:52.386 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-23T11:50:52.386 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-23T11:50:52.386 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.00020901 s, 2.4 MB/s 2026-02-23T11:50:52.387 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-23T11:50:52.444 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_4 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 994 Links: 1 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-23 11:45:34.258514954 +0000 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-23 11:40:41.825165868 +0000 2026-02-23T11:50:52.502 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-23 11:40:41.825165868 +0000 2026-02-23T11:50:52.503 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-23 11:40:41.825165868 +0000 2026-02-23T11:50:52.503 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-23T11:50:52.565 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-23T11:50:52.565 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-23T11:50:52.565 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.000142405 s, 3.6 MB/s 2026-02-23T11:50:52.566 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-23T11:50:52.622 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:50:52.622 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial127.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:50:52.623 INFO:tasks.ceph:Generating config... 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-23T11:50:52.624 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-23T11:50:52.625 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] auth client required = none 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] auth service required = none 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] auth supported = none 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-23T11:50:52.625 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-23T11:50:52.626 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-23T11:50:52.627 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-23T11:50:52.628 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-23T11:50:52.628 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-23T11:50:52.628 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-23T11:50:52.628 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-23T11:50:52.628 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-23T11:50:52.628 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-23T11:50:52.628 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-23T11:50:52.628 INFO:tasks.ceph:Setting up mon.a... 2026-02-23T11:50:52.628 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-23T11:50:52.702 INFO:teuthology.orchestra.run.trial127.stdout:creating /etc/ceph/ceph.keyring 2026-02-23T11:50:52.705 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-23T11:50:52.791 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:50:52.815 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.127')] 2026-02-23T11:50:52.816 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.127', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-23T11:50:52.818 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:52.818 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:50:52.872 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.127 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:50:52.954 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:50:52.952+0000 7f354b03ef40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:50:52.954 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:50:52.952+0000 7f354b03ef40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:50:52.954 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:50:52.952+0000 7f354b03ef40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:50:52.955 INFO:teuthology.orchestra.run.trial127.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:monmaptool: generated fsid 1e2458b0-608e-41bb-b63b-9fe55b3b5a52 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:setting min_mon_release = tentacle 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:epoch 0 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:fsid 1e2458b0-608e-41bb-b63b-9fe55b3b5a52 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:last_changed 2026-02-23T11:50:52.954513+0000 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:created 2026-02-23T11:50:52.954513+0000 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:min_mon_release 20 (tentacle) 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:election_strategy: 1 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:0: [v2:10.20.193.127:3300/0,v1:10.20.193.127:6789/0] mon.a 2026-02-23T11:50:52.956 INFO:teuthology.orchestra.run.trial127.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-23T11:50:52.957 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:50:53.011 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1e2458b0-608e-41bb-b63b-9fe55b3b5a52... 2026-02-23T11:50:53.013 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-23T11:50:53.091 INFO:teuthology.orchestra.run.trial127.stdout:[global] 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: chdir = "" 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: auth supported = none 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: filestore xattr use omap = true 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: mon clock drift allowed = 1.000 2026-02-23T11:50:53.092 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: osd crush chooseleaf type = 0 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: auth debug = true 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: ms die on old message = true 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: ms die on bug = true 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: mon pg warn max object skew = 0 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: # disable pg_autoscaler by default for new pools 2026-02-23T11:50:53.093 INFO:teuthology.orchestra.run.trial127.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: osd pool default size = 2 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon osd allow primary affinity = true 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon osd allow pg remap = true 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on legacy crush tunables = false 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on crush straw calc version zero = false 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on no sortbitwise = false 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on osd down out interval zero = false 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on too few osds = false 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-23T11:50:53.094 INFO:teuthology.orchestra.run.trial127.stdout: mon_allow_pool_size_one = true 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: osd default data pool replay window = 5 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: mon allow pool delete = true 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: mon cluster log file level = debug 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: debug asserts on shutdown = true 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: mon health detail to clog = false 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: mon host = 10.20.193.127 2026-02-23T11:50:53.095 INFO:teuthology.orchestra.run.trial127.stdout: auth client required = none 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: auth cluster required = none 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: auth service required = none 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug asok = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug auth = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug bdev = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug bluefs = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug bluestore = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug buffer = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug civetweb = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug client = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug compressor = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug context = 0/0 2026-02-23T11:50:53.096 INFO:teuthology.orchestra.run.trial127.stdout: debug crush = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug crypto = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug dpdk = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug eventtrace = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug filer = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug filestore = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug finisher = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug fuse = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug heartbeatmap = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug javaclient = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug journal = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug journaler = 0/0 2026-02-23T11:50:53.097 INFO:teuthology.orchestra.run.trial127.stdout: debug kstore = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug leveldb = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug lockdep = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mds = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mds balancer = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mds locker = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mds log = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mds log expire = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mds migrator = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug memdb = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mgr = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mgrc = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug mon = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug monc = 0/0 2026-02-23T11:50:53.098 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug objclass = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug objectcacher = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug objecter = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug optracker = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug osd = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug paxos = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug perfcounter = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug rados = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug rbd = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug rbd mirror = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug rbd replay = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug refs = 0/0 2026-02-23T11:50:53.099 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw lifecycle = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw notification = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw sync = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug rocksdb = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug striper = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug throttle = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug timer = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: debug tp = 0/0 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: ms client mode = crc 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: ms cluster mode = crc 2026-02-23T11:50:53.100 INFO:teuthology.orchestra.run.trial127.stdout: ms mon client mode = crc 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: ms mon cluster mode = crc 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: ms mon service mode = crc 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: ms service mode = crc 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: fsid = 1e2458b0-608e-41bb-b63b-9fe55b3b5a52 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout:[osd] 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: osd journal size = 100 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: osd scrub load threshold = 5.0 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: osd scrub max interval = 600 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: osd mclock profile = high_recovery_ops 2026-02-23T11:50:53.101 INFO:teuthology.orchestra.run.trial127.stdout: osd mclock skip benchmark = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd recover clone overlap = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd recovery max chunk = 1048576 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd debug shutdown = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd debug op order = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd debug verify stray on activate = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd debug trim objects = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd open classes on start = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd debug pg log writeout = true 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.102 INFO:teuthology.orchestra.run.trial127.stdout: osd deep scrub update digest min age = 30 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: osd map max advance = 10 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: journal zero on create = true 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: filestore ondisk finisher threads = 3 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: filestore apply finisher threads = 3 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: bdev debug aio = true 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: osd debug misdirected ops = true 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: debug monc = 20 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: crimson cpu num = 4 2026-02-23T11:50:53.103 INFO:teuthology.orchestra.run.trial127.stdout: crimson osd obc lru size = 10 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug filestore = 0/0 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug journal = 0/0 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 0/0 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug osd = 0/0 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore backref = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore cache = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore cleaner = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore device = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore epm = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore fixedkv tree = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore journal = 20 2026-02-23T11:50:53.104 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore lba = 20 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore odata = 20 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore omap = 20 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore onode = 20 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore t = 20 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore tm = 20 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: osd objectstore = seastore 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: osd shutdown pgref assert = True 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: seastore cachepin size pershard = 64M 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: seastore cachepin type = 2Q 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: seastore max concurrent transactions = 8 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.105 INFO:teuthology.orchestra.run.trial127.stdout:[mgr] 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 1 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug mgr = 20 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug mon = 20 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug auth = 20 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: mgr/telemetry/nag = false 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout:[mon] 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 0/0 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug mon = 0/0 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug paxos = 0/0 2026-02-23T11:50:53.106 INFO:teuthology.orchestra.run.trial127.stdout: debug auth = 20 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: mon data avail warn = 5 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: mon mgr mkfs grace = 240 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: mon osd reporter subtree level = osd 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: mon osd prime pg temp = true 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: auth mon ticket ttl = 660 # 11m 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: auth service ticket ttl = 240 # 4m 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.107 INFO:teuthology.orchestra.run.trial127.stdout: # don't complain about insecure global_id in the test suite 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: # 1m isn't quite enough 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: mon_down_mkfs_grace = 2m 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_filestore_osds = false 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: osd pool default crimson = True 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout:[client] 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: rgw cache enabled = true 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: rgw enable ops log = true 2026-02-23T11:50:53.108 INFO:teuthology.orchestra.run.trial127.stdout: rgw enable usage log = true 2026-02-23T11:50:53.109 INFO:teuthology.orchestra.run.trial127.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-23T11:50:53.109 INFO:teuthology.orchestra.run.trial127.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-23T11:50:53.109 INFO:teuthology.orchestra.run.trial127.stdout:[mon.a] 2026-02-23T11:50:53.110 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-23T11:50:53.110 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-23T11:50:53.199 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-23T11:50:53.200 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:53.200 DEBUG:teuthology.orchestra.run.trial127:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-23T11:50:53.216 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:53.216 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-23T11:50:53.273 INFO:tasks.ceph:Sending monmap to node ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:50:53.273 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:53.273 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-23T11:50:53.273 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:50:53.347 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:50:53.348 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:50:53.405 INFO:tasks.ceph:Setting up mon nodes... 2026-02-23T11:50:53.405 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-23T11:50:53.406 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:50:53.495 INFO:teuthology.orchestra.run.trial127.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:50:53.498 INFO:tasks.ceph:Setting up mds nodes... 2026-02-23T11:50:53.498 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-23T11:50:53.499 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-23T11:50:53.539 INFO:teuthology.orchestra.run.trial127.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-23T11:50:53.552 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-23T11:50:53.552 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial127.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:50:53.552 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-23T11:50:53.615 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:50:53.616 INFO:tasks.ceph:role: osd.0 2026-02-23T11:50:53.616 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:50:53.616 DEBUG:teuthology.orchestra.run.trial127:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-23T11:50:53.680 INFO:teuthology.orchestra.run.trial127.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:50:53.680 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:50:53.680 INFO:teuthology.orchestra.run.trial127.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:50:53.680 INFO:teuthology.orchestra.run.trial127.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:50:53.681 INFO:teuthology.orchestra.run.trial127.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:50:53.681 INFO:teuthology.orchestra.run.trial127.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:50:53.681 INFO:teuthology.orchestra.run.trial127.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:50:53.681 INFO:teuthology.orchestra.run.trial127.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:50:53.681 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:50:53.681 INFO:teuthology.orchestra.run.trial127.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:50:54.262 INFO:teuthology.orchestra.run.trial127.stdout:Discarding blocks...Done. 2026-02-23T11:50:54.264 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial127.front.sepia.ceph.com -o noatime 2026-02-23T11:50:54.264 DEBUG:teuthology.orchestra.run.trial127:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-23T11:50:54.333 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-23T11:50:54.400 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-23T11:50:54.466 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:50:54.466 INFO:tasks.ceph:role: osd.1 2026-02-23T11:50:54.467 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:50:54.467 DEBUG:teuthology.orchestra.run.trial127:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-23T11:50:54.531 INFO:teuthology.orchestra.run.trial127.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:50:54.532 INFO:teuthology.orchestra.run.trial127.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:50:55.093 INFO:teuthology.orchestra.run.trial127.stdout:Discarding blocks...Done. 2026-02-23T11:50:55.096 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial127.front.sepia.ceph.com -o noatime 2026-02-23T11:50:55.096 DEBUG:teuthology.orchestra.run.trial127:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-23T11:50:55.164 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-23T11:50:55.233 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-23T11:50:55.299 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:50:55.299 INFO:tasks.ceph:role: osd.2 2026-02-23T11:50:55.299 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:50:55.300 DEBUG:teuthology.orchestra.run.trial127:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-23T11:50:55.364 INFO:teuthology.orchestra.run.trial127.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:50:55.364 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:50:55.364 INFO:teuthology.orchestra.run.trial127.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:50:55.365 INFO:teuthology.orchestra.run.trial127.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:50:55.365 INFO:teuthology.orchestra.run.trial127.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:50:55.365 INFO:teuthology.orchestra.run.trial127.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:50:55.365 INFO:teuthology.orchestra.run.trial127.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:50:55.365 INFO:teuthology.orchestra.run.trial127.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:50:55.365 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:50:55.365 INFO:teuthology.orchestra.run.trial127.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:50:55.921 INFO:teuthology.orchestra.run.trial127.stdout:Discarding blocks...Done. 2026-02-23T11:50:55.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial127.front.sepia.ceph.com -o noatime 2026-02-23T11:50:55.923 DEBUG:teuthology.orchestra.run.trial127:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-23T11:50:55.992 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-23T11:50:56.058 DEBUG:teuthology.orchestra.run.trial127:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:50:56.209 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:50:56.263 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:50:56,263 seastar - Reactor backend: linux-aio 2026-02-23T11:50:56.273 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:50:56,273 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:50:56.273 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:50:56,273 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:50:56.275 INFO:teuthology.orchestra.run.trial127.stderr:==38806==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:50:56.334 INFO:teuthology.orchestra.run.trial127.stderr:DEBUG 2026-02-23 11:50:56,335 [shard 0:main] seastar - reactor::drain 2026-02-23T11:50:58.370 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:50:58.370 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-23T11:50:58.370 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-23T11:50:58.370 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-23T11:50:58.370 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:50:58.370 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:50:58.389 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:50:58.389 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:50:58.431 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:50:58.431 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:50:58.437 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:50:58,438 seastar - Reactor backend: linux-aio 2026-02-23T11:50:58.452 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:50:58,452 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:50:58.452 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:50:58,452 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:50:58.456 INFO:teuthology.orchestra.run.trial127.stderr:==38804==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:50:58.456 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:50:58,456 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:50:58.599 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x1091317 0xf2e3ce 0x9c906c3 0x9c90cb0 0x9c91033 0x9ca118e 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:50:58.600 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x10910f5 0xf2e3ce 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b8006f 0x9b805bc 0x9b807bb 0x9c9578b 0x9c95ee6 0x9ca0178 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:50:58.600 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x316ca9f 0x9b7e336 0x9b7e5fe 0x9b7ebc0 0x9b83ce5 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:50:58.750 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfdf2c /lib64/libasan.so.8+0xc6d7 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9a48dbb 0x9c7db9c 0x9c7dcca 0x9c9b8a5 0x9c9c048 0x9ca6515 0x77f3d8d 0x7be8169 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:50:58.750 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfee8b /lib64/libasan.so.8+0xfd21f /lib64/libasan.so.8+0xec7e9 0xcb0505 0x9ca658a 0x77f3d8d 0x7be876e 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:50:58.751 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x9b7e39c 0x9b8006f 0x9c939a0 0x9c95346 0x9c959a0 0x9c95ee6 0x9ca0178 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be9638 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:00.899 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:00.899 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-23T11:51:00.900 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-23T11:51:00.900 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-23T11:51:00.900 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:00.900 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:00.931 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T11:51:00.956 DEBUG:teuthology.orchestra.run.trial127:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:51:01.110 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:51:01.166 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:01,166 seastar - Reactor backend: linux-aio 2026-02-23T11:51:01.184 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:01,184 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:01.184 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:01,184 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:01.186 INFO:teuthology.orchestra.run.trial127.stderr:==38876==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:51:01.245 INFO:teuthology.orchestra.run.trial127.stderr:DEBUG 2026-02-23 11:51:01,245 [shard 0:main] seastar - reactor::drain 2026-02-23T11:51:03.279 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:03.279 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-23T11:51:03.280 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-23T11:51:03.280 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-23T11:51:03.280 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:03.280 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:03.299 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:51:03.299 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:51:03.344 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:51:03.344 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:51:03.351 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:03,351 seastar - Reactor backend: linux-aio 2026-02-23T11:51:03.361 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:03,361 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:03.361 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:03,361 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:03.364 INFO:teuthology.orchestra.run.trial127.stderr:==38874==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:51:03.364 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:03,365 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:51:03.504 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x157e6 0x9b88a9b 0x9b891fb 0x9c953b2 0x9c959a0 0x9c95ee6 0x9ca0178 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:03.504 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x316ca9f 0x9b7e336 0x9b8006f 0x9b805bc 0x9b807bb 0x9c9578b 0x9c95ee6 0x9ca0178 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:03.504 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x9b7e2e6 0x9b8006f 0x9b805bc 0x9b807bb 0x9c9578b 0x9c95ee6 0x9ca0178 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:03.653 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a5bbdd 0x9b87e24 0x9c7cb87 0x9c7cd4f 0x9c87c4b 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:03.653 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe1d5af 0x9b03de9 0x9c864c2 0x9c865bb 0x7be81a6 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:03.653 INFO:teuthology.orchestra.run.trial127.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9ca91052 0xffffffff9d74d451 0xffffffff9d800ba2 2026-02-23T11:51:03.653 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d8 /lib64/libasan.so.8+0x15197 0x1125e3c 0xf2e3c3 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:05.793 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:05.793 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-23T11:51:05.793 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-23T11:51:05.793 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-23T11:51:05.793 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:05.793 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:05.824 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T11:51:05.848 DEBUG:teuthology.orchestra.run.trial127:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:51:05.999 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:51:06.052 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:06,052 seastar - Reactor backend: linux-aio 2026-02-23T11:51:06.067 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:06,067 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:06.067 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:06,067 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:06.069 INFO:teuthology.orchestra.run.trial127.stderr:==38942==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:51:06.128 INFO:teuthology.orchestra.run.trial127.stderr:DEBUG 2026-02-23 11:51:06,128 [shard 0:main] seastar - reactor::drain 2026-02-23T11:51:08.178 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:08.178 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-23T11:51:08.178 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-23T11:51:08.179 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-23T11:51:08.179 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:08.179 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:08.198 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:51:08.198 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:51:08.243 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:51:08.243 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:51:08.250 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:08,250 seastar - Reactor backend: linux-aio 2026-02-23T11:51:08.260 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:08,260 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:08.260 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:08,260 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:51:08.264 INFO:teuthology.orchestra.run.trial127.stderr:==38940==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:51:08.264 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-23 11:51:08,264 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:51:08.405 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0xe0756b 0x7be8061 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:08.405 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x1125e3c 0xf2e3c3 0x9c906c3 0x9c90cb0 0x9c91033 0x9ca118e 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:08.405 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc4d1 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9a48dbb 0x9b02622 0x9b02738 0x9b02814 0x9b02a90 0x9b03037 0x9b0343b 0x9b036f8 0x9ca27bf 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:08.551 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15225 0x1125e3c 0xf2e3c3 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b8006f 0x9b805bc 0x9b807bb 0x9c9578b 0x9c95ee6 0x9ca0178 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:08.551 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0xe069e3 0x1125de5 0xf2e3c3 0x9c9096b 0x9c90cb0 0x9c91033 0x9ca118e 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:08.551 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1586f 0x35eac7b 0x35eadfb 0x3606c78 0x9b8b6b3 0x9b8b775 0x9c872e7 0x9c8747c 0x9c8780e 0x9c87a19 0x9c87d55 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:51:10.677 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:10.677 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-23T11:51:10.677 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-23T11:51:10.677 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-23T11:51:10.677 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-23T11:51:10.677 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:10.708 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T11:51:10.733 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-23T11:51:10.734 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:10.734 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-23T11:51:10.798 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:10.798 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-23T11:51:10.862 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:10.862 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-23T11:51:10.926 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:10.926 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-23T11:51:10.991 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:10.991 DEBUG:teuthology.orchestra.run.trial127:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-23T11:51:11.047 INFO:tasks.ceph:Adding keys to all mons... 2026-02-23T11:51:11.047 DEBUG:teuthology.orchestra.run.trial127:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-23T11:51:11.109 INFO:teuthology.orchestra.run.trial127.stdout:[mgr.x] 2026-02-23T11:51:11.109 INFO:teuthology.orchestra.run.trial127.stdout: key = AQAdP5xplLOMHRAAHWv3M55jIANEO/6UEYJ36Q== 2026-02-23T11:51:11.109 INFO:teuthology.orchestra.run.trial127.stdout:[osd.0] 2026-02-23T11:51:11.110 INFO:teuthology.orchestra.run.trial127.stdout: key = AQAiP5xpjJ1OHxAAYaqc3eNrG3xfcOJ87GsSxQ== 2026-02-23T11:51:11.110 INFO:teuthology.orchestra.run.trial127.stdout:[osd.1] 2026-02-23T11:51:11.110 INFO:teuthology.orchestra.run.trial127.stdout: key = AQAnP5xp6IqcGRAAjpXLB85HCDpN2QRD72GOOw== 2026-02-23T11:51:11.110 INFO:teuthology.orchestra.run.trial127.stdout:[osd.2] 2026-02-23T11:51:11.110 INFO:teuthology.orchestra.run.trial127.stdout: key = AQAsP5xpUoyzExAAz2/9zNw/M14NlRaunklqWw== 2026-02-23T11:51:11.110 INFO:teuthology.orchestra.run.trial127.stdout:[client.0] 2026-02-23T11:51:11.110 INFO:teuthology.orchestra.run.trial127.stdout: key = AQAdP5xpc4kvIBAAq+4phuzbk8fLvtCrnFT/kA== 2026-02-23T11:51:11.111 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-23T11:51:11.198 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:51:11.247 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:51:11.296 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:51:11.344 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-23T11:51:11.393 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-23T11:51:11.394 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-23T11:51:11.418 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-23T11:51:11.504 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.503+0000 7f43cf2c6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.505 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.503+0000 7f43cf2c6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.512 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-23T11:51:11.537 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:51:11.593 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-23T11:51:11.594 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-23T11:51:11.594 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-23T11:51:11.635 INFO:tasks.ceph.mon.a:Started 2026-02-23T11:51:11.635 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-23T11:51:11.636 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-23T11:51:11.636 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-23T11:51:11.638 INFO:tasks.ceph.mgr.x:Started 2026-02-23T11:51:11.638 DEBUG:tasks.ceph:set 0 configs 2026-02-23T11:51:11.639 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph config dump 2026-02-23T11:51:11.698 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-23T11:51:11.696+0000 7ff6b0e92c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.698 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-23T11:51:11.696+0000 7ff6b0e92c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.698 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-23T11:51:11.696+0000 7ff6b0e92c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.702 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-23T11:51:11.700+0000 7f02315cbfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.714 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.711+0000 7ff7f6a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.716 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.715+0000 7ff7f6a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.747 INFO:teuthology.orchestra.run.trial127.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-23T11:51:11.755 INFO:tasks.ceph:Setting crush tunables to default 2026-02-23T11:51:11.755 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd crush tunables default 2026-02-23T11:51:11.829 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.824+0000 7f54d9fe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.830 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.829+0000 7f54d9fe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.863 INFO:teuthology.orchestra.run.trial127.stderr:adjusted tunables profile to default 2026-02-23T11:51:11.871 INFO:tasks.ceph:check_enable_crimson: True 2026-02-23T11:51:11.871 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-23T11:51:11.871 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-23T11:51:11.905 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-23T11:51:11.904+0000 7f02315cbfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.905 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-23T11:51:11.904+0000 7f02315cbfc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.944 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.940+0000 7f4782d6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.945 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:11.944+0000 7f4782d6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:11.983 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-23T11:51:11.982+0000 7ff6a85f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-23T11:51:11.983 INFO:tasks.ceph.mon.a.trial127.stderr:Please be aware that this feature is experimental, untested, 2026-02-23T11:51:11.983 INFO:tasks.ceph.mon.a.trial127.stderr:unsupported, and may result in data corruption, data loss, 2026-02-23T11:51:11.983 INFO:tasks.ceph.mon.a.trial127.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-23T11:51:11.983 INFO:tasks.ceph.mon.a.trial127.stderr:feature with important data. 2026-02-23T11:51:11.983 INFO:tasks.ceph.mon.a.trial127.stderr: 2026-02-23T11:51:12.734 INFO:tasks.ceph.mgr.x.trial127.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T11:51:12.735 INFO:tasks.ceph.mgr.x.trial127.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T11:51:12.735 INFO:tasks.ceph.mgr.x.trial127.stderr: from numpy import show_config as show_numpy_config 2026-02-23T11:51:12.873 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-23T11:51:12.874 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:12.874 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-23T11:51:12.938 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:12.938 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-23T11:51:13.002 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:13.002 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-23T11:51:13.066 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd new 17434fd0-1d5e-4e42-91ef-a9ec187563ad 0 2026-02-23T11:51:13.180 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.176+0000 7f2022a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.181 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.180+0000 7f2022a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.218 INFO:teuthology.orchestra.run.trial127.stdout:0 2026-02-23T11:51:13.227 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd new 2602c60a-c762-48f4-80f2-cbfafd6c0ce1 1 2026-02-23T11:51:13.302 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.297+0000 7f938a30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.303 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.302+0000 7f938a30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.340 INFO:teuthology.orchestra.run.trial127.stdout:1 2026-02-23T11:51:13.348 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd new d3a94156-b27f-49ec-896a-c1b02fbc9780 2 2026-02-23T11:51:13.422 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.418+0000 7f26d1e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.423 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.422+0000 7f26d1e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.461 INFO:teuthology.orchestra.run.trial127.stdout:2 2026-02-23T11:51:13.469 DEBUG:teuthology.orchestra.run.trial127:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-23T11:51:13.496 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-23T11:51:13.497 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-23T11:51:13.538 INFO:tasks.ceph.osd.0:Started 2026-02-23T11:51:13.538 DEBUG:teuthology.orchestra.run.trial127:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-23T11:51:13.567 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-23T11:51:13.567 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-23T11:51:13.612 INFO:tasks.ceph.osd.1:Started 2026-02-23T11:51:13.613 DEBUG:teuthology.orchestra.run.trial127:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-23T11:51:13.644 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-23T11:51:13.644 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-23T11:51:13.691 INFO:tasks.ceph.osd.2:Started 2026-02-23T11:51:13.692 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:51:13.770 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.767+0000 7fc07bacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.771 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:13.770+0000 7fc07bacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:13.804 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:13.804 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":6,"fsid":"1e2458b0-608e-41bb-b63b-9fe55b3b5a52","created":"2026-02-23T11:51:11.709063+0000","modified":"2026-02-23T11:51:13.461447+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:51:13.461447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"17434fd0-1d5e-4e42-91ef-a9ec187563ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2602c60a-c762-48f4-80f2-cbfafd6c0ce1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3a94156-b27f-49ec-896a-c1b02fbc9780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:51:13.813 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-23T11:51:13.813 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-23T11:51:14.115 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:51:14.183 INFO:teuthology.misc.health.trial127.stderr:2026-02-23T11:51:14.180+0000 7f90538cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:14.185 INFO:teuthology.misc.health.trial127.stderr:2026-02-23T11:51:14.183+0000 7f90538cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:14.215 INFO:teuthology.misc.health.trial127.stdout: 2026-02-23T11:51:14.215 INFO:teuthology.misc.health.trial127.stdout:{"epoch":6,"fsid":"1e2458b0-608e-41bb-b63b-9fe55b3b5a52","created":"2026-02-23T11:51:11.709063+0000","modified":"2026-02-23T11:51:13.461447+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:51:13.461447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"17434fd0-1d5e-4e42-91ef-a9ec187563ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2602c60a-c762-48f4-80f2-cbfafd6c0ce1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3a94156-b27f-49ec-896a-c1b02fbc9780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:51:14.222 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:51:14.951 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-23T11:51:14.950+0000 7f014e57f640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-23T11:51:20.524 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:51:20.591 INFO:teuthology.misc.health.trial127.stderr:2026-02-23T11:51:20.589+0000 7efee4d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:20.593 INFO:teuthology.misc.health.trial127.stderr:2026-02-23T11:51:20.592+0000 7efee4d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:20.702 INFO:teuthology.misc.health.trial127.stdout: 2026-02-23T11:51:20.703 INFO:teuthology.misc.health.trial127.stdout:{"epoch":7,"fsid":"1e2458b0-608e-41bb-b63b-9fe55b3b5a52","created":"2026-02-23T11:51:11.709063+0000","modified":"2026-02-23T11:51:19.948256+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:51:13.461447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"17434fd0-1d5e-4e42-91ef-a9ec187563ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2602c60a-c762-48f4-80f2-cbfafd6c0ce1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3a94156-b27f-49ec-896a-c1b02fbc9780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:51:20.710 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:51:27.012 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:51:27.083 INFO:teuthology.misc.health.trial127.stderr:2026-02-23T11:51:27.080+0000 7feca3011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:27.084 INFO:teuthology.misc.health.trial127.stderr:2026-02-23T11:51:27.084+0000 7feca3011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:27.192 INFO:teuthology.misc.health.trial127.stdout: 2026-02-23T11:51:27.193 INFO:teuthology.misc.health.trial127.stdout:{"epoch":13,"fsid":"1e2458b0-608e-41bb-b63b-9fe55b3b5a52","created":"2026-02-23T11:51:11.709063+0000","modified":"2026-02-23T11:51:26.964046+0000","last_up_change":"2026-02-23T11:51:22.955646+0000","last_in_change":"2026-02-23T11:51:13.461447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:51:23.962921+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17434fd0-1d5e-4e42-91ef-a9ec187563ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6806","nonce":1545382719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6805","nonce":1545382719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6811","nonce":1545382719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6810","nonce":1545382719}]},"public_addr":"10.20.193.127:6806/1545382719","cluster_addr":"10.20.193.127:6805/1545382719","heartbeat_back_addr":"10.20.193.127:6811/1545382719","heartbeat_front_addr":"10.20.193.127:6810/1545382719","state":["exists","up"]},{"osd":1,"uuid":"2602c60a-c762-48f4-80f2-cbfafd6c0ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6808","nonce":2035584209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6809","nonce":2035584209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6812","nonce":2035584209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6813","nonce":2035584209}]},"public_addr":"10.20.193.127:6808/2035584209","cluster_addr":"10.20.193.127:6809/2035584209","heartbeat_back_addr":"10.20.193.127:6812/2035584209","heartbeat_front_addr":"10.20.193.127:6813/2035584209","state":["exists","up"]},{"osd":2,"uuid":"d3a94156-b27f-49ec-896a-c1b02fbc9780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6804","nonce":1631685259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6803","nonce":1631685259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6807","nonce":1631685259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6802","nonce":1631685259}]},"public_addr":"10.20.193.127:6804/1631685259","cluster_addr":"10.20.193.127:6803/1631685259","heartbeat_back_addr":"10.20.193.127:6807/1631685259","heartbeat_front_addr":"10.20.193.127:6802/1631685259","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:51:27.201 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-23T11:51:27.201 INFO:tasks.ceph:Creating RBD pool 2026-02-23T11:51:27.201 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-23T11:51:27.274 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:27.271+0000 7ff669e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:27.275 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:27.275+0000 7ff669e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:27.971 INFO:teuthology.orchestra.run.trial127.stderr:pool 'rbd' created 2026-02-23T11:51:27.984 DEBUG:teuthology.orchestra.run.trial127:> rbd --cluster ceph pool init rbd 2026-02-23T11:51:28.024 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:28.023+0000 7f55c9ae0240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:28.024 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:28.024+0000 7f55c9ae0240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:28.025 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:28.025+0000 7f55c9ae0240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.008 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-23T11:51:31.009 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-23T11:51:31.009 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-23T11:51:31.125 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.122+0000 7f3e2b25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.126 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.126+0000 7f3e2b25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.236 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:31.245 INFO:teuthology.orchestra.run.trial127.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-23T11:51:31.245 INFO:tasks.ceph_manager:config epoch is 1 2026-02-23T11:51:31.245 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-23T11:51:31.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-23T11:51:31.246 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-23T11:51:31.322 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.320+0000 7fed59783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.323 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.323+0000 7fed59783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.464 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:31.471 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6800","nonce":1406200625},{"type":"v1","addr":"10.20.193.127:6801","nonce":1406200625}]},"active_addr":"10.20.193.127:6801/1406200625","active_change":"2026-02-23T11:51:13.946277+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":2379065174}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":1958937796}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":445806006}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":4094324003}]}]} 2026-02-23T11:51:31.476 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-23T11:51:31.476 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-23T11:51:31.476 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:51:31.552 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.550+0000 7f2b959a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.553 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.553+0000 7f2b959a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.663 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:31.663 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":17,"fsid":"1e2458b0-608e-41bb-b63b-9fe55b3b5a52","created":"2026-02-23T11:51:11.709063+0000","modified":"2026-02-23T11:51:30.979697+0000","last_up_change":"2026-02-23T11:51:22.955646+0000","last_in_change":"2026-02-23T11:51:13.461447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:51:23.962921+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T11:51:27.387439+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17434fd0-1d5e-4e42-91ef-a9ec187563ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6806","nonce":1545382719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6805","nonce":1545382719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6811","nonce":1545382719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6810","nonce":1545382719}]},"public_addr":"10.20.193.127:6806/1545382719","cluster_addr":"10.20.193.127:6805/1545382719","heartbeat_back_addr":"10.20.193.127:6811/1545382719","heartbeat_front_addr":"10.20.193.127:6810/1545382719","state":["exists","up"]},{"osd":1,"uuid":"2602c60a-c762-48f4-80f2-cbfafd6c0ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6808","nonce":2035584209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6809","nonce":2035584209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6812","nonce":2035584209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6813","nonce":2035584209}]},"public_addr":"10.20.193.127:6808/2035584209","cluster_addr":"10.20.193.127:6809/2035584209","heartbeat_back_addr":"10.20.193.127:6812/2035584209","heartbeat_front_addr":"10.20.193.127:6813/2035584209","state":["exists","up"]},{"osd":2,"uuid":"d3a94156-b27f-49ec-896a-c1b02fbc9780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6804","nonce":1631685259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6803","nonce":1631685259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6807","nonce":1631685259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6802","nonce":1631685259}]},"public_addr":"10.20.193.127:6804/1631685259","cluster_addr":"10.20.193.127:6803/1631685259","heartbeat_back_addr":"10.20.193.127:6807/1631685259","heartbeat_front_addr":"10.20.193.127:6802/1631685259","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:51:31.672 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-23T11:51:31.672 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:51:31.747 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.745+0000 7f1ca61b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.748 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.748+0000 7f1ca61b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.858 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:31.859 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":17,"fsid":"1e2458b0-608e-41bb-b63b-9fe55b3b5a52","created":"2026-02-23T11:51:11.709063+0000","modified":"2026-02-23T11:51:30.979697+0000","last_up_change":"2026-02-23T11:51:22.955646+0000","last_in_change":"2026-02-23T11:51:13.461447+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:51:23.962921+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T11:51:27.387439+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17434fd0-1d5e-4e42-91ef-a9ec187563ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6806","nonce":1545382719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6805","nonce":1545382719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6811","nonce":1545382719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6810","nonce":1545382719}]},"public_addr":"10.20.193.127:6806/1545382719","cluster_addr":"10.20.193.127:6805/1545382719","heartbeat_back_addr":"10.20.193.127:6811/1545382719","heartbeat_front_addr":"10.20.193.127:6810/1545382719","state":["exists","up"]},{"osd":1,"uuid":"2602c60a-c762-48f4-80f2-cbfafd6c0ce1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6808","nonce":2035584209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6809","nonce":2035584209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6812","nonce":2035584209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6813","nonce":2035584209}]},"public_addr":"10.20.193.127:6808/2035584209","cluster_addr":"10.20.193.127:6809/2035584209","heartbeat_back_addr":"10.20.193.127:6812/2035584209","heartbeat_front_addr":"10.20.193.127:6813/2035584209","state":["exists","up"]},{"osd":2,"uuid":"d3a94156-b27f-49ec-896a-c1b02fbc9780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6804","nonce":1631685259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6803","nonce":1631685259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6807","nonce":1631685259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6802","nonce":1631685259}]},"public_addr":"10.20.193.127:6804/1631685259","cluster_addr":"10.20.193.127:6803/1631685259","heartbeat_back_addr":"10.20.193.127:6807/1631685259","heartbeat_front_addr":"10.20.193.127:6802/1631685259","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:51:31.869 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-23T11:51:31.870 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-23T11:51:31.870 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-23T11:51:31.952 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.949+0000 7f04ba974640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.952 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.949+0000 7f222d5fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.953 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.949+0000 7f2b9aa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.953 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.953+0000 7f2b9aa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.954 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.953+0000 7f222d5fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.954 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:31.953+0000 7f04ba974640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:31.968 INFO:teuthology.orchestra.run.trial127.stdout:42949672968 2026-02-23T11:51:31.968 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:51:31.969 INFO:teuthology.orchestra.run.trial127.stdout:38654705673 2026-02-23T11:51:31.970 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:51:31.971 INFO:teuthology.orchestra.run.trial127.stdout:42949672968 2026-02-23T11:51:31.971 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:51:32.089 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:32.086+0000 7f5a0801a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:32.089 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:32.086+0000 7f7568f83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:32.090 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:32.086+0000 7f76b9ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:32.090 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:32.090+0000 7f5a0801a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:32.091 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:32.091+0000 7f7568f83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:32.091 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:32.091+0000 7f76b9ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:32.203 INFO:teuthology.orchestra.run.trial127.stdout:38654705668 2026-02-23T11:51:32.208 INFO:teuthology.orchestra.run.trial127.stdout:42949672968 2026-02-23T11:51:32.208 INFO:teuthology.orchestra.run.trial127.stdout:42949672968 2026-02-23T11:51:32.215 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705668 for osd.2 2026-02-23T11:51:32.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.0 2026-02-23T11:51:32.220 DEBUG:teuthology.parallel:result is None 2026-02-23T11:51:32.221 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.1 2026-02-23T11:51:32.221 DEBUG:teuthology.parallel:result is None 2026-02-23T11:51:33.216 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:51:33.293 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:33.291+0000 7f23d2acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:33.294 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:33.294+0000 7f23d2acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:33.403 INFO:teuthology.orchestra.run.trial127.stdout:38654705668 2026-02-23T11:51:33.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705668 for osd.2 2026-02-23T11:51:34.414 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:51:34.492 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:34.489+0000 7f914505a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:34.493 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:34.493+0000 7f914505a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:34.603 INFO:teuthology.orchestra.run.trial127.stdout:38654705673 2026-02-23T11:51:34.613 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.2 2026-02-23T11:51:34.613 DEBUG:teuthology.parallel:result is None 2026-02-23T11:51:34.613 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T11:51:34.614 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:51:34.690 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:34.687+0000 7f43d290b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:34.691 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:34.691+0000 7f43d290b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:34.801 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:34.802 INFO:teuthology.orchestra.run.trial127.stderr:dumped all 2026-02-23T11:51:34.811 INFO:teuthology.orchestra.run.trial127.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T11:51:33.950323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917044,"kb_used_data":62917044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369308,"statfs":{"total":161061224448,"available":96634171392,"internally_reserved":0,"allocated":64427053056,"data_stored":64427053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.988310"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.995420+0000","last_change":"2026-02-23T11:51:30.998040+0000","last_active":"2026-02-23T11:51:30.995420+0000","last_peered":"2026-02-23T11:51:30.995420+0000","last_clean":"2026-02-23T11:51:30.995420+0000","last_became_active":"2026-02-23T11:51:29.005723+0000","last_became_peered":"2026-02-23T11:51:29.005723+0000","last_unstale":"2026-02-23T11:51:30.995420+0000","last_undegraded":"2026-02-23T11:51:30.995420+0000","last_fullsized":"2026-02-23T11:51:30.995420+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996546+0000","last_change":"2026-02-23T11:51:31.000141+0000","last_active":"2026-02-23T11:51:30.996546+0000","last_peered":"2026-02-23T11:51:30.996546+0000","last_clean":"2026-02-23T11:51:30.996546+0000","last_became_active":"2026-02-23T11:51:29.004440+0000","last_became_peered":"2026-02-23T11:51:29.004440+0000","last_unstale":"2026-02-23T11:51:30.996546+0000","last_undegraded":"2026-02-23T11:51:30.996546+0000","last_fullsized":"2026-02-23T11:51:30.996546+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996562+0000","last_change":"2026-02-23T11:51:30.999323+0000","last_active":"2026-02-23T11:51:30.996562+0000","last_peered":"2026-02-23T11:51:30.996562+0000","last_clean":"2026-02-23T11:51:30.996562+0000","last_became_active":"2026-02-23T11:51:29.012562+0000","last_became_peered":"2026-02-23T11:51:29.012562+0000","last_unstale":"2026-02-23T11:51:30.996562+0000","last_undegraded":"2026-02-23T11:51:30.996562+0000","last_fullsized":"2026-02-23T11:51:30.996562+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996487+0000","last_change":"2026-02-23T11:51:31.000166+0000","last_active":"2026-02-23T11:51:30.996487+0000","last_peered":"2026-02-23T11:51:30.996487+0000","last_clean":"2026-02-23T11:51:30.996487+0000","last_became_active":"2026-02-23T11:51:29.011000+0000","last_became_peered":"2026-02-23T11:51:29.011000+0000","last_unstale":"2026-02-23T11:51:30.996487+0000","last_undegraded":"2026-02-23T11:51:30.996487+0000","last_fullsized":"2026-02-23T11:51:30.996487+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996348+0000","last_change":"2026-02-23T11:51:30.996348+0000","last_active":"2026-02-23T11:51:30.996348+0000","last_peered":"2026-02-23T11:51:30.996348+0000","last_clean":"2026-02-23T11:51:30.996348+0000","last_became_active":"2026-02-23T11:51:29.013840+0000","last_became_peered":"2026-02-23T11:51:29.013840+0000","last_unstale":"2026-02-23T11:51:30.996348+0000","last_undegraded":"2026-02-23T11:51:30.996348+0000","last_fullsized":"2026-02-23T11:51:30.996348+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.993517+0000","last_change":"2026-02-23T11:51:30.995523+0000","last_active":"2026-02-23T11:51:30.993517+0000","last_peered":"2026-02-23T11:51:30.993517+0000","last_clean":"2026-02-23T11:51:30.993517+0000","last_became_active":"2026-02-23T11:51:29.025629+0000","last_became_peered":"2026-02-23T11:51:29.025629+0000","last_unstale":"2026-02-23T11:51:30.993517+0000","last_undegraded":"2026-02-23T11:51:30.993517+0000","last_fullsized":"2026-02-23T11:51:30.993517+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.992295+0000","last_change":"2026-02-23T11:51:30.994981+0000","last_active":"2026-02-23T11:51:30.992295+0000","last_peered":"2026-02-23T11:51:30.992295+0000","last_clean":"2026-02-23T11:51:30.992295+0000","last_became_active":"2026-02-23T11:51:29.021361+0000","last_became_peered":"2026-02-23T11:51:29.021361+0000","last_unstale":"2026-02-23T11:51:30.992295+0000","last_undegraded":"2026-02-23T11:51:30.992295+0000","last_fullsized":"2026-02-23T11:51:30.992295+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996392+0000","last_change":"2026-02-23T11:51:30.998719+0000","last_active":"2026-02-23T11:51:30.996392+0000","last_peered":"2026-02-23T11:51:30.996392+0000","last_clean":"2026-02-23T11:51:30.996392+0000","last_became_active":"2026-02-23T11:51:29.005664+0000","last_became_peered":"2026-02-23T11:51:29.005664+0000","last_unstale":"2026-02-23T11:51:30.996392+0000","last_undegraded":"2026-02-23T11:51:30.996392+0000","last_fullsized":"2026-02-23T11:51:30.996392+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.995489+0000","last_change":"2026-02-23T11:51:25.992914+0000","last_active":"2026-02-23T11:51:30.995489+0000","last_peered":"2026-02-23T11:51:30.995489+0000","last_clean":"2026-02-23T11:51:30.995489+0000","last_became_active":"2026-02-23T11:51:25.985543+0000","last_became_peered":"2026-02-23T11:51:25.985543+0000","last_unstale":"2026-02-23T11:51:30.995489+0000","last_undegraded":"2026-02-23T11:51:30.995489+0000","last_fullsized":"2026-02-23T11:51:30.995489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:24.957489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:24.957489+0000","last_clean_scrub_stamp":"2026-02-23T11:51:24.957489+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:51:34.812 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:51:34.889 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:34.887+0000 7f77c61dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:34.890 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:34.890+0000 7f77c61dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:34.999 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:35.000 INFO:teuthology.orchestra.run.trial127.stderr:dumped all 2026-02-23T11:51:35.010 INFO:teuthology.orchestra.run.trial127.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T11:51:33.950323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917044,"kb_used_data":62917044,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369308,"statfs":{"total":161061224448,"available":96634171392,"internally_reserved":0,"allocated":64427053056,"data_stored":64427053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.988310"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.995420+0000","last_change":"2026-02-23T11:51:30.998040+0000","last_active":"2026-02-23T11:51:30.995420+0000","last_peered":"2026-02-23T11:51:30.995420+0000","last_clean":"2026-02-23T11:51:30.995420+0000","last_became_active":"2026-02-23T11:51:29.005723+0000","last_became_peered":"2026-02-23T11:51:29.005723+0000","last_unstale":"2026-02-23T11:51:30.995420+0000","last_undegraded":"2026-02-23T11:51:30.995420+0000","last_fullsized":"2026-02-23T11:51:30.995420+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996546+0000","last_change":"2026-02-23T11:51:31.000141+0000","last_active":"2026-02-23T11:51:30.996546+0000","last_peered":"2026-02-23T11:51:30.996546+0000","last_clean":"2026-02-23T11:51:30.996546+0000","last_became_active":"2026-02-23T11:51:29.004440+0000","last_became_peered":"2026-02-23T11:51:29.004440+0000","last_unstale":"2026-02-23T11:51:30.996546+0000","last_undegraded":"2026-02-23T11:51:30.996546+0000","last_fullsized":"2026-02-23T11:51:30.996546+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996562+0000","last_change":"2026-02-23T11:51:30.999323+0000","last_active":"2026-02-23T11:51:30.996562+0000","last_peered":"2026-02-23T11:51:30.996562+0000","last_clean":"2026-02-23T11:51:30.996562+0000","last_became_active":"2026-02-23T11:51:29.012562+0000","last_became_peered":"2026-02-23T11:51:29.012562+0000","last_unstale":"2026-02-23T11:51:30.996562+0000","last_undegraded":"2026-02-23T11:51:30.996562+0000","last_fullsized":"2026-02-23T11:51:30.996562+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996487+0000","last_change":"2026-02-23T11:51:31.000166+0000","last_active":"2026-02-23T11:51:30.996487+0000","last_peered":"2026-02-23T11:51:30.996487+0000","last_clean":"2026-02-23T11:51:30.996487+0000","last_became_active":"2026-02-23T11:51:29.011000+0000","last_became_peered":"2026-02-23T11:51:29.011000+0000","last_unstale":"2026-02-23T11:51:30.996487+0000","last_undegraded":"2026-02-23T11:51:30.996487+0000","last_fullsized":"2026-02-23T11:51:30.996487+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996348+0000","last_change":"2026-02-23T11:51:30.996348+0000","last_active":"2026-02-23T11:51:30.996348+0000","last_peered":"2026-02-23T11:51:30.996348+0000","last_clean":"2026-02-23T11:51:30.996348+0000","last_became_active":"2026-02-23T11:51:29.013840+0000","last_became_peered":"2026-02-23T11:51:29.013840+0000","last_unstale":"2026-02-23T11:51:30.996348+0000","last_undegraded":"2026-02-23T11:51:30.996348+0000","last_fullsized":"2026-02-23T11:51:30.996348+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.993517+0000","last_change":"2026-02-23T11:51:30.995523+0000","last_active":"2026-02-23T11:51:30.993517+0000","last_peered":"2026-02-23T11:51:30.993517+0000","last_clean":"2026-02-23T11:51:30.993517+0000","last_became_active":"2026-02-23T11:51:29.025629+0000","last_became_peered":"2026-02-23T11:51:29.025629+0000","last_unstale":"2026-02-23T11:51:30.993517+0000","last_undegraded":"2026-02-23T11:51:30.993517+0000","last_fullsized":"2026-02-23T11:51:30.993517+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.992295+0000","last_change":"2026-02-23T11:51:30.994981+0000","last_active":"2026-02-23T11:51:30.992295+0000","last_peered":"2026-02-23T11:51:30.992295+0000","last_clean":"2026-02-23T11:51:30.992295+0000","last_became_active":"2026-02-23T11:51:29.021361+0000","last_became_peered":"2026-02-23T11:51:29.021361+0000","last_unstale":"2026-02-23T11:51:30.992295+0000","last_undegraded":"2026-02-23T11:51:30.992295+0000","last_fullsized":"2026-02-23T11:51:30.992295+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.996392+0000","last_change":"2026-02-23T11:51:30.998719+0000","last_active":"2026-02-23T11:51:30.996392+0000","last_peered":"2026-02-23T11:51:30.996392+0000","last_clean":"2026-02-23T11:51:30.996392+0000","last_became_active":"2026-02-23T11:51:29.005664+0000","last_became_peered":"2026-02-23T11:51:29.005664+0000","last_unstale":"2026-02-23T11:51:30.996392+0000","last_undegraded":"2026-02-23T11:51:30.996392+0000","last_fullsized":"2026-02-23T11:51:30.996392+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:27.967530+0000","last_clean_scrub_stamp":"2026-02-23T11:51:27.967530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:51:30.995489+0000","last_change":"2026-02-23T11:51:25.992914+0000","last_active":"2026-02-23T11:51:30.995489+0000","last_peered":"2026-02-23T11:51:30.995489+0000","last_clean":"2026-02-23T11:51:30.995489+0000","last_became_active":"2026-02-23T11:51:25.985543+0000","last_became_peered":"2026-02-23T11:51:25.985543+0000","last_unstale":"2026-02-23T11:51:30.995489+0000","last_undegraded":"2026-02-23T11:51:30.995489+0000","last_fullsized":"2026-02-23T11:51:30.995489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:51:24.957489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:51:24.957489+0000","last_clean_scrub_stamp":"2026-02-23T11:51:24.957489+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:51:35.011 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T11:51:35.012 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-23T11:51:35.012 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-23T11:51:35.012 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-23T11:51:35.088 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:35.086+0000 7f19411dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:35.089 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:51:35.089+0000 7f19411dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:51:35.224 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:35.225 INFO:teuthology.orchestra.run.trial127.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-23T11:51:35.235 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-23T11:51:35.235 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-23T11:51:35.321 DEBUG:teuthology.orchestra.run.trial127:> echo 'StrictHostKeyChecking no 2026-02-23T11:51:35.321 DEBUG:teuthology.orchestra.run.trial127:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-23T11:51:35.337 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCjf9izvDN/P06SfC05JhzPkh0g9z3AElWR5K4d6iQya+U+xfhchEFCl/Jkns0rxB7Nr14Jpa8wykp8vxLTc5D0eb+bWcckZbFax6Q0IIVNscrRXyj3uJ6kxr8fhVsEfLUNhEkIeMogOS7U6Q1i7IK78dKhBxKl10BYM8H+CVkzigPbJb+Ad8AKWsdVieHOpxk7YsHIhdI90N+x5P68x70hYCUK05CnI6GUznVy/Ssgx3IPdqAdSEERiTIO3CFoCLwH5kaxNZnPaLU2BTUBL1QqrpQeV29KOpaTnIHLPda0QCZSY4vGt4uth7SZZy2UIfGTytbn6GRx6oGkGOjqh0G9 2026-02-23T11:51:35.337 INFO:teuthology.task.ssh_keys:pushing keys to trial127.front.sepia.ceph.com for ubuntu 2026-02-23T11:51:35.337 DEBUG:teuthology.orchestra.run.trial127:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-23T11:51:35.394 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:35.394 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-23T11:51:35.394 DEBUG:teuthology.orchestra.run.trial127:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-23T11:51:35.450 DEBUG:teuthology.orchestra.run.trial127:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-23T11:51:35.507 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:35.507 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-23T11:51:35.563 DEBUG:teuthology.orchestra.run.trial127:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-23_11:51:35:563582 2026-02-23T11:51:35.620 INFO:teuthology.orchestra.run.trial127.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-23_11:51:35:563582' 2026-02-23T11:51:35.620 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:35.621 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-23T11:51:35.676 INFO:teuthology.run_tasks:Running task cbt... 2026-02-23T11:51:35.701 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial127.front.sepia.ceph.com', 'osds': ['ubuntu@trial127.front.sepia.ceph.com'], 'mons': ['ubuntu@trial127.front.sepia.ceph.com'], 'clients': ['ubuntu@trial127.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial127.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-23T11:51:35.701 DEBUG:teuthology.orchestra.run.trial127:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-23T11:51:35.736 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:51:35.736 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-23T11:51:35.788 DEBUG:teuthology.orchestra.run.trial127:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-23T11:51:35.851 INFO:teuthology.orchestra.run.trial127.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-23T11:51:36.083 INFO:tasks.cbt:Installing collectl 2026-02-23T11:51:36.083 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-23T11:51:36.158 INFO:teuthology.orchestra.run.trial127.stderr:--2026-02-23 11:51:36-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-23T11:51:36.210 INFO:teuthology.orchestra.run.trial127.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:c95, ... 2026-02-23T11:51:36.213 INFO:teuthology.orchestra.run.trial127.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-23T11:51:36.411 INFO:teuthology.orchestra.run.trial127.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-23T11:51:36.412 INFO:teuthology.orchestra.run.trial127.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnD9IsYTE2oi8KPzy2kg5aA9oNo7GA3hicQTBm-PKOsmOc-YjbvbrNknbpbCR1ctE3cG-pdgUVIl8zzxw8PGmx5VcOg%3D%3D&use_mirror=phoenixnap&r= [following] 2026-02-23T11:51:36.412 INFO:teuthology.orchestra.run.trial127.stderr:--2026-02-23 11:51:36-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnD9IsYTE2oi8KPzy2kg5aA9oNo7GA3hicQTBm-PKOsmOc-YjbvbrNknbpbCR1ctE3cG-pdgUVIl8zzxw8PGmx5VcOg%3D%3D&use_mirror=phoenixnap&r= 2026-02-23T11:51:36.467 INFO:teuthology.orchestra.run.trial127.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-23T11:51:36.471 INFO:teuthology.orchestra.run.trial127.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-23T11:51:36.644 INFO:teuthology.orchestra.run.trial127.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-23T11:51:36.644 INFO:teuthology.orchestra.run.trial127.stderr:Location: https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-23T11:51:36.644 INFO:teuthology.orchestra.run.trial127.stderr:--2026-02-23 11:51:36-- https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-23T11:51:36.650 INFO:teuthology.orchestra.run.trial127.stderr:Resolving phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)... 184.164.141.26 2026-02-23T11:51:36.713 INFO:teuthology.orchestra.run.trial127.stderr:Connecting to phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)|184.164.141.26|:443... connected. 2026-02-23T11:51:38.606 INFO:teuthology.orchestra.run.trial127.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-23T11:51:38.606 INFO:teuthology.orchestra.run.trial127.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-23T11:51:38.606 INFO:teuthology.orchestra.run.trial127.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-23T11:51:38.606 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:41.142 INFO:teuthology.orchestra.run.trial127.stderr: 0K .......... .......... .......... .......... .......... 8% 19.7K 28s 2026-02-23T11:51:41.205 INFO:teuthology.orchestra.run.trial127.stderr: 50K .......... .......... .......... .......... .......... 16% 805K 13s 2026-02-23T11:51:41.281 INFO:teuthology.orchestra.run.trial127.stderr: 100K .......... .......... .......... .......... .......... 24% 674K 8s 2026-02-23T11:51:41.281 INFO:teuthology.orchestra.run.trial127.stderr: 150K .......... .......... .......... .......... .......... 32% 129M 5s 2026-02-23T11:51:41.344 INFO:teuthology.orchestra.run.trial127.stderr: 200K .......... .......... .......... .......... .......... 41% 814K 4s 2026-02-23T11:51:41.344 INFO:teuthology.orchestra.run.trial127.stderr: 250K .......... .......... .......... .......... .......... 49% 163M 3s 2026-02-23T11:51:41.344 INFO:teuthology.orchestra.run.trial127.stderr: 300K .......... .......... .......... .......... .......... 57% 105M 2s 2026-02-23T11:51:41.344 INFO:teuthology.orchestra.run.trial127.stderr: 350K .......... .......... .......... .......... .......... 65% 66.0M 1s 2026-02-23T11:51:41.403 INFO:teuthology.orchestra.run.trial127.stderr: 400K .......... .......... .......... .......... .......... 74% 829K 1s 2026-02-23T11:51:43.985 INFO:teuthology.orchestra.run.trial127.stderr: 450K .......... .......... .......... .......... .......... 82% 19.4K 1s 2026-02-23T11:51:44.047 INFO:teuthology.orchestra.run.trial127.stderr: 500K .......... .......... .......... .......... .......... 90% 805K 1s 2026-02-23T11:51:44.109 INFO:teuthology.orchestra.run.trial127.stderr: 550K .......... .......... .......... .......... .......... 98% 808K 0s 2026-02-23T11:51:44.109 INFO:teuthology.orchestra.run.trial127.stderr: 600K ...... 100% 11.4T=5.5s 2026-02-23T11:51:44.110 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:44.110 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23 11:51:44 (110 KB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-23T11:51:44.110 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:51:44.133 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/ 2026-02-23T11:51:44.133 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/collectl.conf 2026-02-23T11:51:44.133 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/ 2026-02-23T11:51:44.133 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Socket.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/colmux.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Import.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/style.css 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-23T11:51:44.134 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Process.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Export.html 2026-02-23T11:51:44.135 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Startup.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Performance.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Messages.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Logging.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Misc.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Examples.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-23T11:51:44.136 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Playback.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Hello.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Network.html 2026-02-23T11:51:44.137 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Memory.html 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Disks.html 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/service/ 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/service/collectl.service 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/ 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/misc.ph 2026-02-23T11:51:44.138 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/gexpr.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/statsd.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/envrules.std 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/hello.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/vmsum.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/proctree.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/README-WINDOWS 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/formatit.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/colmux 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/INSTALL 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/graphite.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/vmstat.ph 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/README 2026-02-23T11:51:44.139 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/man1/ 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/man1/collectl.1 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/man1/colmux.1 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/lexpr.ph 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/client.pl 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/collectl 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/COPYING 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/vnet.ph 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/GPL 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/UNINSTALL 2026-02-23T11:51:44.140 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/ARTISTIC 2026-02-23T11:51:44.175 INFO:teuthology.orchestra.run.trial127.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-23T11:51:44.177 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-23T11:51:44.526 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:01:05 ago on Mon 23 Feb 2026 11:50:39 AM UTC. 2026-02-23T11:51:44.612 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout: librbd-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 24 k 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout: perf x86_64 5.14.0-681.el9 appstream 3.2 M 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout: libradospp-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 31 k 2026-02-23T11:51:44.613 INFO:teuthology.orchestra.run.trial127.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout:Install 6 Packages 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 3.7 M 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 14 M 2026-02-23T11:51:44.614 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-23T11:51:45.192 INFO:teuthology.orchestra.run.trial127.stdout:(1/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 4.7 MB/s | 290 kB 00:00 2026-02-23T11:51:45.472 INFO:teuthology.orchestra.run.trial127.stdout:(2/6): perf-5.14.0-681.el9.x86_64.rpm 11 MB/s | 3.2 MB 00:00 2026-02-23T11:51:45.484 INFO:teuthology.orchestra.run.trial127.stdout:(3/6): libradospp-devel-20.3.0-5439.ge371433f.e 89 kB/s | 31 kB 00:00 2026-02-23T11:51:45.488 INFO:teuthology.orchestra.run.trial127.stdout:(4/6): librbd-devel-20.3.0-5439.ge371433f.el9.x 67 kB/s | 24 kB 00:00 2026-02-23T11:51:45.541 INFO:teuthology.orchestra.run.trial127.stdout:(5/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 229 kB/s | 13 kB 00:00 2026-02-23T11:51:45.586 INFO:teuthology.orchestra.run.trial127.stdout:(6/6): pdsh-2.34-7.el9.x86_64.rpm 1.0 MB/s | 118 kB 00:00 2026-02-23T11:51:45.588 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:51:45.588 INFO:teuthology.orchestra.run.trial127.stdout:Total 3.7 MB/s | 3.7 MB 00:00 2026-02-23T11:51:45.612 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-23T11:51:45.622 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-23T11:51:45.622 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-23T11:51:45.677 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-23T11:51:45.678 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-23T11:51:45.749 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-23T11:51:45.754 INFO:teuthology.orchestra.run.trial127.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-23T11:51:45.795 INFO:teuthology.orchestra.run.trial127.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-23T11:51:45.799 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-23T11:51:45.801 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libradospp-devel-2:20.3.0-5439.ge371433f.el9.x86_64 4/6 2026-02-23T11:51:45.962 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librbd-devel-2:20.3.0-5439.ge371433f.el9.x86_64 5/6 2026-02-23T11:51:45.983 INFO:teuthology.orchestra.run.trial127.stdout: Installing : perf-5.14.0-681.el9.x86_64 6/6 2026-02-23T11:51:46.089 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: perf-5.14.0-681.el9.x86_64 6/6 2026-02-23T11:51:46.089 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libradospp-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/6 2026-02-23T11:51:46.089 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librbd-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/6 2026-02-23T11:51:46.089 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-23T11:51:46.089 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : perf-5.14.0-681.el9.x86_64 4/6 2026-02-23T11:51:46.089 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: libradospp-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: librbd-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: pdsh-2.34-7.el9.x86_64 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-23T11:51:46.150 INFO:teuthology.orchestra.run.trial127.stdout: perf-5.14.0-681.el9.x86_64 2026-02-23T11:51:46.151 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-23T11:51:46.151 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-23T11:51:46.182 DEBUG:teuthology.orchestra.run.trial127:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-23T11:51:46.319 INFO:teuthology.orchestra.run.trial127.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-23T11:51:46.328 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-23T11:51:46.785 INFO:teuthology.orchestra.run.trial127.stdout:Collecting lxml 2026-02-23T11:51:46.816 INFO:teuthology.orchestra.run.trial127.stdout: Downloading lxml-6.0.2-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.3 MB) 2026-02-23T11:51:47.179 INFO:teuthology.orchestra.run.trial127.stdout:Collecting matplotlib 2026-02-23T11:51:47.184 INFO:teuthology.orchestra.run.trial127.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-23T11:51:47.227 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-23T11:51:47.261 INFO:teuthology.orchestra.run.trial127.stdout:Collecting importlib-resources>=3.2.0 2026-02-23T11:51:47.266 INFO:teuthology.orchestra.run.trial127.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-23T11:51:47.586 INFO:teuthology.orchestra.run.trial127.stdout:Collecting fonttools>=4.22.0 2026-02-23T11:51:47.590 INFO:teuthology.orchestra.run.trial127.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-23T11:51:47.622 INFO:teuthology.orchestra.run.trial127.stdout:Collecting cycler>=0.10 2026-02-23T11:51:47.626 INFO:teuthology.orchestra.run.trial127.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-23T11:51:47.627 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-23T11:51:47.627 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-23T11:51:47.775 INFO:teuthology.orchestra.run.trial127.stdout:Collecting contourpy>=1.0.1 2026-02-23T11:51:47.779 INFO:teuthology.orchestra.run.trial127.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-23T11:51:47.782 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-23T11:51:47.932 INFO:teuthology.orchestra.run.trial127.stdout:Collecting kiwisolver>=1.3.1 2026-02-23T11:51:47.936 INFO:teuthology.orchestra.run.trial127.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-23T11:51:48.343 INFO:teuthology.orchestra.run.trial127.stdout:Collecting pillow>=8 2026-02-23T11:51:48.348 INFO:teuthology.orchestra.run.trial127.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-23T11:51:48.453 INFO:teuthology.orchestra.run.trial127.stdout:Collecting zipp>=3.1.0 2026-02-23T11:51:48.456 INFO:teuthology.orchestra.run.trial127.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-23T11:51:48.467 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-23T11:51:48.580 INFO:teuthology.orchestra.run.trial127.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib, lxml 2026-02-23T11:51:49.707 INFO:teuthology.orchestra.run.trial127.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 lxml-6.0.2 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-23T11:51:49.824 INFO:tasks.cbt:benchmark: radosbench 2026-02-23T11:51:49.825 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-23T11:51:49.852 DEBUG:teuthology.orchestra.run.trial127:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr:11:51:49 - DEBUG - cbt - Settings.cluster: 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'clients': ['ubuntu@trial127.front.sepia.ceph.com'], 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'head': 'ubuntu@trial127.front.sepia.ceph.com', 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'is_teuthology': True, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'iterations': 1, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'mons': ['ubuntu@trial127.front.sepia.ceph.com'], 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'osds': ['ubuntu@trial127.front.sepia.ceph.com'], 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'osds_per_node': 3, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'pid_dir': '/var/run/ceph', 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'pgp_size': 256, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'replication': 'replicated'}}, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'rebuild_every_test': False, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'rgws': {'ubuntu@trial127.front.sepia.ceph.com': None}, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'use_existing': True, 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr: 'user': 'ubuntu'} 2026-02-23T11:51:49.966 INFO:teuthology.orchestra.run.trial127.stderr:11:51:49 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-23T11:51:49.967 INFO:teuthology.orchestra.run.trial127.stderr:11:51:49 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-23T11:51:49.967 INFO:teuthology.orchestra.run.trial127.stderr:11:51:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-23T11:51:50.136 INFO:teuthology.orchestra.run.trial127.stderr:11:51:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-23T11:51:50.138 INFO:teuthology.orchestra.run.trial127.stderr:11:51:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2026-02-23T11:51:50.305 INFO:teuthology.orchestra.run.trial127.stderr:11:51:50 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-23T11:51:50.306 INFO:teuthology.orchestra.run.trial127.stderr:11:51:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-23T11:52:50.363 INFO:teuthology.orchestra.run.trial127.stderr:11:52:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-23T11:52:50.536 INFO:teuthology.orchestra.run.trial127.stderr:11:52:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-23T11:52:50.722 INFO:teuthology.orchestra.run.trial127.stderr:11:52:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-23T11:52:50.724 INFO:teuthology.orchestra.run.trial127.stderr:11:52:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-23T11:52:50.726 INFO:teuthology.orchestra.run.trial127.stderr:11:52:50 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* 2026-02-23T11:52:50.909 INFO:teuthology.orchestra.run.trial127.stderr:11:52:50 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial127.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-23T11:52:51.067 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo killall -9 rados 2026-02-23T11:52:51.242 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-23T11:52:51.242 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3. 2026-02-23T11:52:51.242 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-23T11:52:51.243 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-23T11:52:51.243 INFO:teuthology.orchestra.run.trial127.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'readmode': 'rand', 'time': 60, 'write_only': False, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-23T11:52:51.243 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-23T11:52:51.243 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-23T11:52:51.243 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-23T11:52:51.244 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-23T11:52:51.422 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-23T11:52:51.578 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-23T11:52:51.580 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2026-02-23T11:52:51.746 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-23T11:52:51.746 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-23T11:52:51.748 INFO:teuthology.orchestra.run.trial127.stderr:11:52:51 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-23T11:52:52.088 INFO:teuthology.orchestra.run.trial127.stderr:11:52:52 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-23T11:52:53.046 INFO:teuthology.orchestra.run.trial127.stderr:11:52:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-23T11:52:54.046 INFO:teuthology.orchestra.run.trial127.stderr:11:52:54 - INFO - cbt - Checking Health after pool creation. 2026-02-23T11:52:59.051 INFO:teuthology.orchestra.run.trial127.stderr:11:52:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-23T11:52:59.402 INFO:teuthology.orchestra.run.trial127.stderr:11:52:59 - INFO - cbt - Final Pool Health Check. 2026-02-23T11:53:04.403 INFO:teuthology.orchestra.run.trial127.stderr:11:53:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-23T11:53:04.758 INFO:teuthology.orchestra.run.trial127.stderr:11:53:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-23T11:53:04.938 INFO:teuthology.orchestra.run.trial127.stderr:11:53:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-23T11:53:05.123 INFO:teuthology.orchestra.run.trial127.stderr:11:53:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-23T11:53:05.124 INFO:teuthology.orchestra.run.trial127.stderr:11:53:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-23T11:53:05.126 INFO:teuthology.orchestra.run.trial127.stderr:11:53:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sync 2026-02-23T11:53:05.291 INFO:teuthology.orchestra.run.trial127.stderr:11:53:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-23T11:53:06.247 INFO:teuthology.orchestra.run.trial127.stderr:11:53:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-23T11:53:06.459 INFO:teuthology.orchestra.run.trial127.stderr:11:53:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write 2026-02-23T11:53:06.619 INFO:teuthology.orchestra.run.trial127.stderr:11:53:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2026-02-23T11:53:06.899 INFO:teuthology.orchestra.run.trial127.stderr:11:53:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl 2026-02-23T11:53:07.058 INFO:teuthology.orchestra.run.trial127.stderr:11:53:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-23T11:53:07.059 INFO:teuthology.orchestra.run.trial127.stderr:11:53:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf 2026-02-23T11:53:07.219 INFO:teuthology.orchestra.run.trial127.stderr:11:53:07 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-23T11:53:07.220 INFO:teuthology.orchestra.run.trial127.stderr:11:53:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-23T11:53:07.221 INFO:teuthology.orchestra.run.trial127.stderr:11:53:07 - INFO - cbt - Running radosbench write test. 2026-02-23T11:53:07.221 INFO:teuthology.orchestra.run.trial127.stderr:11:53:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.0 2026-02-23T11:53:07.222 INFO:teuthology.orchestra.run.trial127.stderr:11:53:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.1 2026-02-23T11:54:08.437 INFO:teuthology.orchestra.run.trial127.stderr:11:54:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-23T11:54:08.616 INFO:teuthology.orchestra.run.trial127.stderr:11:54:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-23T11:54:08.797 INFO:teuthology.orchestra.run.trial127.stderr:11:54:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2026-02-23T11:54:08.798 INFO:teuthology.orchestra.run.trial127.stderr:11:54:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-23T11:54:08.800 INFO:teuthology.orchestra.run.trial127.stderr:11:54:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2026-02-23T11:54:09.152 INFO:teuthology.orchestra.run.trial127.stderr:11:54:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* 2026-02-23T11:54:09.333 INFO:teuthology.orchestra.run.trial127.stderr:11:54:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial127.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write 2026-02-23T11:54:09.495 INFO:teuthology.orchestra.run.trial127.stderr:11:54:09 - INFO - cbt - Convert results to json format. 2026-02-23T11:54:09.495 INFO:teuthology.orchestra.run.trial127.stderr:11:54:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sync 2026-02-23T11:54:09.784 INFO:teuthology.orchestra.run.trial127.stderr:11:54:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-23T11:54:10.265 INFO:teuthology.orchestra.run.trial127.stderr:11:54:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-23T11:54:10.478 INFO:teuthology.orchestra.run.trial127.stderr:11:54:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand 2026-02-23T11:54:10.639 INFO:teuthology.orchestra.run.trial127.stderr:11:54:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/ceph_settings.out 2026-02-23T11:54:10.944 INFO:teuthology.orchestra.run.trial127.stderr:11:54:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/collectl 2026-02-23T11:54:11.106 INFO:teuthology.orchestra.run.trial127.stderr:11:54:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-23T11:54:11.108 INFO:teuthology.orchestra.run.trial127.stderr:11:54:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf 2026-02-23T11:54:11.273 INFO:teuthology.orchestra.run.trial127.stderr:11:54:11 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-23T11:54:11.273 INFO:teuthology.orchestra.run.trial127.stderr:11:54:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.${pid} & done 2026-02-23T11:54:11.275 INFO:teuthology.orchestra.run.trial127.stderr:11:54:11 - INFO - cbt - Running radosbench rand test. 2026-02-23T11:54:11.275 INFO:teuthology.orchestra.run.trial127.stderr:11:54:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/output.0 2026-02-23T11:54:11.277 INFO:teuthology.orchestra.run.trial127.stderr:11:54:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/output.1 2026-02-23T11:55:12.481 INFO:teuthology.orchestra.run.trial127.stderr:11:55:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-23T11:55:12.655 INFO:teuthology.orchestra.run.trial127.stderr:11:55:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-23T11:55:12.836 INFO:teuthology.orchestra.run.trial127.stderr:11:55:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf.data 2026-02-23T11:55:12.837 INFO:teuthology.orchestra.run.trial127.stderr:11:55:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.* 2026-02-23T11:55:12.839 INFO:teuthology.orchestra.run.trial127.stderr:11:55:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/historic_ops.out 2026-02-23T11:55:13.203 INFO:teuthology.orchestra.run.trial127.stderr:11:55:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/* 2026-02-23T11:55:13.372 INFO:teuthology.orchestra.run.trial127.stderr:11:55:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial127.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand 2026-02-23T11:55:13.531 INFO:teuthology.orchestra.run.trial127.stderr:11:55:13 - INFO - cbt - Convert results to json format. 2026-02-23T11:55:13.542 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-23T11:55:13.552 DEBUG:teuthology.orchestra.run.trial127:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-23T11:55:13.578 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-23T11:55:13.578 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-23T11:55:13.578 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-23T11:55:13.637 INFO:teuthology.orchestra.run.trial127.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.trial127.front.sepia.ceph.com 2026-02-23T11:55:13.638 INFO:teuthology.orchestra.run.trial127.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.trial127.front.sepia.ceph.com 2026-02-23T11:55:13.638 INFO:teuthology.orchestra.run.trial127.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.trial127.front.sepia.ceph.com 2026-02-23T11:55:13.638 INFO:teuthology.orchestra.run.trial127.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.trial127.front.sepia.ceph.com 2026-02-23T11:55:13.640 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:55:13.640 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.trial127.front.sepia.ceph.com of=/dev/stdout 2026-02-23T11:55:13.695 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-23T11:55:13.695 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2026-02-23T11:55:13.751 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-23T11:55:13.751 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-23T11:55:13.751 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:55:13.751 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.trial127.front.sepia.ceph.com of=/dev/stdout 2026-02-23T11:55:13.807 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-23T11:55:13.807 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2026-02-23T11:55:13.863 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-23T11:55:13.863 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-23T11:55:13.863 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:55:13.863 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.trial127.front.sepia.ceph.com of=/dev/stdout 2026-02-23T11:55:13.919 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-23T11:55:13.920 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2026-02-23T11:55:13.976 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-23T11:55:13.976 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-23T11:55:13.977 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-23T11:55:13.977 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.trial127.front.sepia.ceph.com of=/dev/stdout 2026-02-23T11:55:14.033 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-23T11:55:14.033 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2026-02-23T11:55:14.089 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-23T11:55:14.089 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-23T11:55:14.093 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-23T11:55:14.093 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-23T11:55:14.093 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-23T11:55:14.105 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-23T11:55:14.106 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-23T11:55:14.106 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial127.front.sepia.ceph.com 2026-02-23T11:55:14.106 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-23T11:55:14.145 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-23T11:55:14.202 DEBUG:teuthology.orchestra.run.trial127:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-23T11:55:14.260 INFO:teuthology.orchestra.run.trial127.stdout:"600" 2026-02-23T11:55:14.260 DEBUG:teuthology.orchestra.run.trial127:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-23_11:51:35:563582 /home/ubuntu/.ssh/authorized_keys 2026-02-23T11:55:14.317 DEBUG:teuthology.orchestra.run.trial127:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-23T11:55:14.374 DEBUG:teuthology.orchestra.run.trial127:> rm /home/ubuntu/.ssh/config 2026-02-23T11:55:14.429 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-23T11:55:14.441 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-23T11:55:14.544 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:55:14.541+0000 7fc291e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:14.545 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-23T11:55:14.545+0000 7fc291e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:14.673 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-23T11:55:14.674 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-23T11:55:14.674 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-23T11:55:14.674 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:55:14.754 INFO:tasks.ceph.osd.0:Stopped 2026-02-23T11:55:14.754 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-23T11:55:14.754 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:55:14.850 INFO:tasks.ceph.osd.1:Stopped 2026-02-23T11:55:14.851 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-23T11:55:14.851 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:55:14.946 INFO:tasks.ceph.osd.2:Stopped 2026-02-23T11:55:14.946 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-23T11:55:14.946 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-23T11:55:14.946 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:55:14.981 INFO:tasks.ceph.mgr.x:Stopped 2026-02-23T11:55:14.981 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-23T11:55:14.981 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-23T11:55:14.981 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:55:15.030 INFO:tasks.ceph.mon.a:Stopped 2026-02-23T11:55:15.031 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-23T11:55:15.031 DEBUG:teuthology.orchestra.run.trial127:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-23T11:55:15.097 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:55:15.097 DEBUG:teuthology.orchestra.run.trial127:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-23T11:55:15.166 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:55:15.167 DEBUG:teuthology.orchestra.run.trial127:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-23T11:55:15.233 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:55:15.233 DEBUG:teuthology.orchestra.run.trial127:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-23T11:55:15.301 INFO:tasks.ceph:Archiving mon data... 2026-02-23T11:55:15.303 DEBUG:teuthology.misc:Transferring archived files from trial127:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66280/data/mon.a.tgz 2026-02-23T11:55:15.303 DEBUG:teuthology.orchestra.run.trial127:> mktemp 2026-02-23T11:55:15.359 INFO:teuthology.orchestra.run.trial127.stdout:/tmp/tmp.XlmDqpKU8I 2026-02-23T11:55:15.360 DEBUG:teuthology.orchestra.run.trial127:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.XlmDqpKU8I 2026-02-23T11:55:15.448 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 0666 /tmp/tmp.XlmDqpKU8I 2026-02-23T11:55:15.529 DEBUG:teuthology.orchestra.remote:trial127:/tmp/tmp.XlmDqpKU8I is 204KB 2026-02-23T11:55:15.583 DEBUG:teuthology.orchestra.run.trial127:> rm -fr /tmp/tmp.XlmDqpKU8I 2026-02-23T11:55:15.600 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-23T11:55:15.600 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-23T11:55:15.722 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-23T11:55:15.722 INFO:tasks.ceph:Archiving crash dumps... 2026-02-23T11:55:15.725 DEBUG:teuthology.misc:Transferring archived files from trial127:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66280/remote/trial127/crash 2026-02-23T11:55:15.726 DEBUG:teuthology.orchestra.run.trial127:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-23T11:55:15.788 INFO:tasks.ceph:Compressing logs... 2026-02-23T11:55:15.788 DEBUG:teuthology.orchestra.run.trial127:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:55:15.855 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-23T11:55:15.856 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-23T11:55:15.856 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2026-02-23T11:55:15.856 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43946.log 2026-02-23T11:55:15.856 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44556.log 2026-02-23T11:55:15.856 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2026-02-23T11:55:15.856 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2026-02-23T11:55:15.856 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.43946.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43946.log.gz 2026-02-23T11:55:15.857 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.44556.log: 91.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-23T11:55:15.857 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43908.log 2026-02-23T11:55:15.857 INFO:teuthology.orchestra.run.trial127.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.44556.log.gz 2026-02-23T11:55:15.857 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-23T11:55:15.857 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2026-02-23T11:55:15.857 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.43908.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43908.log.gz 2026-02-23T11:55:15.868 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45251.log 2026-02-23T11:55:15.868 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39839.log 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.45251.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.45251.log.gz 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.39839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44000.log 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40399.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39839.log.gz 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.44000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43794.log 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44000.log.gz 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.44554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45928.log 2026-02-23T11:55:15.869 INFO:teuthology.orchestra.run.trial127.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.43794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43794.log.gz 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.45928.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45928.log.gz 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40860.log 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40058.log 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40860.log.gz 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.39466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43850.log 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40058.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38171.log 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40058.log.gz 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.43850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log 2026-02-23T11:55:15.870 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43850.log.gz 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph.tmp-client.admin.38171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38171.log.gz 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41150.log 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40513.log.gz 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.41150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41003.log 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41150.log.gz 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45246.log 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.41003.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr: 87.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-23T11:55:15.871 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41003.log.gz 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.45246.log: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.45246.log.gz 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40374.log 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39561.log 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40374.log.gz 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.39561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39561.log.gz 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.39792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40304.log 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.41052.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-23T11:55:15.872 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40464.log 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39745.log 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40304.log.gz 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40714.log 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40464.log.gz 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.39745.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39745.log.gz 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr: 49.6%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.40714.log.gz 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40713.log 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2026-02-23T11:55:15.873 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39608.log 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40713.log.gz 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40611.log 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.39608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40235.log 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39608.log.gz 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40611.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40611.log.gz 2026-02-23T11:55:15.874 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.40235.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40235.log.gz 2026-02-23T11:55:15.875 INFO:teuthology.orchestra.run.trial127.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-23T11:55:21.441 INFO:teuthology.orchestra.run.trial127.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-23T11:55:21.710 INFO:teuthology.orchestra.run.trial127.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-23T11:55:22.178 INFO:teuthology.orchestra.run.trial127.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-23T11:55:22.181 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-23T11:55:22.181 INFO:teuthology.orchestra.run.trial127.stderr:real 0m6.336s 2026-02-23T11:55:22.181 INFO:teuthology.orchestra.run.trial127.stderr:user 0m16.607s 2026-02-23T11:55:22.181 INFO:teuthology.orchestra.run.trial127.stderr:sys 0m1.135s 2026-02-23T11:55:22.182 INFO:tasks.ceph:Archiving logs... 2026-02-23T11:55:22.182 DEBUG:teuthology.misc:Transferring archived files from trial127:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66280/remote/trial127/log 2026-02-23T11:55:22.183 DEBUG:teuthology.orchestra.run.trial127:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-23T11:55:23.695 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:55:23.759 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:55:23.771 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-23T11:55:23.771 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-23T11:55:23.797 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-23T11:55:23.808 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-23T11:55:23.808 DEBUG:teuthology.orchestra.run.trial127:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:55:23.854 INFO:teuthology.orchestra.run.trial127.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:55:23.874 INFO:teuthology.orchestra.run.trial127.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:55:23.874 INFO:teuthology.orchestra.run.trial127.stdout:=============================================================================== 2026-02-23T11:55:23.874 INFO:teuthology.orchestra.run.trial127.stdout:^* 50-203-248-23-static.hfc> 2 6 377 24 +2332us[+2331us] +/- 32ms 2026-02-23T11:55:23.874 INFO:teuthology.orchestra.run.trial127.stdout:^+ 72-14-186-59.ip.linodeus> 2 6 377 23 -878us[ -878us] +/- 91ms 2026-02-23T11:55:23.874 INFO:teuthology.orchestra.run.trial127.stdout:^+ 73.185.182.209 2 6 177 25 +2304us[+2303us] +/- 49ms 2026-02-23T11:55:23.874 INFO:teuthology.orchestra.run.trial127.stdout:^+ 172-104-28-175.ip.linode> 2 6 377 23 -4152us[-4152us] +/- 60ms 2026-02-23T11:55:23.876 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-23T11:55:23.889 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-23T11:55:23.890 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-23T11:55:23.900 DEBUG:teuthology.orchestra.run.trial127:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-23T11:55:23.974 DEBUG:teuthology.orchestra.run.trial127:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:55:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:55:24.000 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-23T11:55:24.010 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-23T11:55:24.021 INFO:teuthology.task.internal:Duration was 1038.047603 seconds 2026-02-23T11:55:24.021 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-23T11:55:24.034 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-23T11:55:24.034 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-23T11:55:24.078 INFO:teuthology.orchestra.run.trial127.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:55:24.461 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-23T11:55:24.461 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial127.front.sepia.ceph.com 2026-02-23T11:55:24.461 DEBUG:teuthology.orchestra.run.trial127:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-23T11:55:24.519 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-23T11:55:24.519 DEBUG:teuthology.orchestra.run.trial127:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:55:24.624 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-23T11:55:24.624 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:55:24.647 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:55:24.648 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:55:24.648 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-23T11:55:24.648 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:55:24.648 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-23T11:55:24.659 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-23T11:55:24.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-23T11:55:24.676 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-23T11:55:24.676 DEBUG:teuthology.orchestra.run.trial127:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-23T11:55:24.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-23T11:55:24.742 DEBUG:teuthology.orchestra.run.trial127:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:55:24.789 INFO:teuthology.orchestra.run.trial127.stdout:kernel.core_pattern = core 2026-02-23T11:55:24.820 DEBUG:teuthology.orchestra.run.trial127:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:55:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:55:24.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-23T11:55:24.894 INFO:teuthology.task.internal:Transferring archived files... 2026-02-23T11:55:24.895 DEBUG:teuthology.misc:Transferring archived files from trial127:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66280/remote/trial127 2026-02-23T11:55:24.896 DEBUG:teuthology.orchestra.run.trial127:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-23T11:55:24.962 INFO:teuthology.task.internal:Removing archive directory... 2026-02-23T11:55:24.962 DEBUG:teuthology.orchestra.run.trial127:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-23T11:55:25.010 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-23T11:55:25.023 INFO:teuthology.task.internal:Not uploading archives. 2026-02-23T11:55:25.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-23T11:55:25.037 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-23T11:55:25.037 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-23T11:55:25.066 INFO:teuthology.orchestra.run.trial127.stdout: 917521 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 11:55 /home/ubuntu/cephtest 2026-02-23T11:55:25.066 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-23T11:55:25.076 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-23T11:55:25.105 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read} duration: 1038.0476033687592 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-23T11:55:25.105 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:55:25.213 INFO:teuthology.run:pass